Have you got yourself into situations where you have a k8s pod using a slim image (no shell) and you need to -it into it to debug the app or maybe do a few internal tests?
Slim images are safer and have less surface for attacks, so not having shell and other binaries makes it hardened in terms of security, but at the same time harder to debug and troubleshoot issues.
My “weapon of choice” these days in order to debug cases like this is to bring up an ephemaral debug containers by issuing the kubectl debug command, this allows you to inspect the shared namespace process, do a full copy of the container and more.
And it’s not just this use case where we can use the debug command, but with nodes and other options as well.
The minimal requirement is to have k8s cluster 1.23 or up version:
Requirements:
Kubernetes 1.23+- ephemeral containers in beta, requires EphemeralContainers feature gateKubernetes 1.25+- ephemeral containers GA, no feature gate required
I’ll be writing a few scenarios on how to use with my minikube with version 1.38+
minikube version
minikube version: v1.38.1
commit: c93a4cb9311efc66b90d33ea03f75f2c4120e9b0
I’ll explore first the debug for a slim container, so no shell, and the shared process namespace when creating a debug ephemeral container.
1. Inspecting a slim image
For this example I’m going to create a new image with just a simple go web app, on port 3000, but the main base image for the binary will be scratch, a debian image with no shell and missing loads of binaries.
Here’s the mininal code to run a simple http server on go:
package main
import (
"log"
"net/http"
)
func healthHandler(w http.ResponseWriter, r *http.Request) {
log.Println("health request!")
w.WriteHeader(http.StatusOK)
}
func main() {
log.Println("starting service ...")
http.Handle("/health", http.HandlerFunc(healthHandler))
panic(http.ListenAndServe(":3000", nil))
}
I can make this simple code part of a Docker multistage build and use the scratch image as the final base image:
FROM golang:1.26 AS builder
COPY <<EOF ./service.go
package main
import (
"log"
"net/http"
)
func healthHandler(w http.ResponseWriter, r *http.Request) {
log.Println("health request!")
w.WriteHeader(http.StatusOK)
}
func main() {
log.Println("starting service ...")
http.Handle("/health", http.HandlerFunc(healthHandler))
panic(http.ListenAndServe(":3000", nil))
}
EOF
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o /bin/service ./service.go
FROM scratch
COPY --from=builder /bin/service /bin/service
CMD ["/bin/service"]
I’m using podman so the build can be done with:
podman build . -t simple-go-service:v1.0.0
Let me test and run the process and test the endpoints:
podman run -d -p 3000:3000 simple-go-service:v1.0.0
And then we can do a few test calls:
curl localhost:3000/health
podman logs -f dazzling_visvesvaraya
2026/05/07 14:56:47 starting service ...
2026/05/07 14:57:24 health request!
Let’s try to do -it on the running container still on podman:
podman exec -ti dazzling_visvesvaraya /bin/bash
And then we get the following on trying to shell into it:
podman exec -ti dazzling_visvesvaraya /bin/bash
Error: crun: executable file `/bin/bash` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found
%
podman exec -ti dazzling_visvesvaraya /bin/sh
Error: crun: executable file `/bin/sh` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found
No shell to interact and go into the container.
In terms of security, these apps and images are great, as there pratically no binaries on it, so less ways to exploit it, but it gets hard when trying to debug / troubleshoot.
So now let me load the image from podman into minikube and create a pod using it, and let’s try to debug and run commands using kubectl debug.
We can use the podman desktop as it has a easier way to upload the images to the cluster, and as I’m using a multinode cluster it’s easier to do it through the UI.
.
It’s mainly generating a file and then uploading to the cluster for building. Check it here.
Let’s check if the image can be listed within minikube:
minikube image ls
minikube image ls
registry.k8s.io/pause:3.10.1
registry.k8s.io/kube-scheduler:v1.35.1
registry.k8s.io/kube-proxy:v1.35.1
registry.k8s.io/kube-controller-manager:v1.35.1
registry.k8s.io/kube-apiserver:v1.35.1
registry.k8s.io/etcd:3.6.6-0
registry.k8s.io/coredns/coredns:v1.13.1
localhost/trivy-webhook:latest
localhost/simple-go-service:v1.0.0
gcr.io/k8s-minikube/storage-provisioner:v5
docker.io/stefanprodan/podinfo:latest
docker.io/library/ubuntu:latest
docker.io/library/ubuntu:latest
docker.io/library/simple-go-service:v1.0.0
docker.io/library/postgres:14.22-trixie
docker.io/library/nginx:latest
docker.io/kindest/kindnetd:v20250512-df8de77b
Let’s create the pod and try to use -it again and try to get into the pod:
kubectl run simple-go-service --image=localhost/simple-go-service:v1.0.0 --restart="Never" --image-pull-policy="Never"
kubectl exec -it simple-go-service -- /bin/bash
kubectl exec -it simple-go-service -- /bin/bash
error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "19466a19aabcb534669f881777d2c9484d2f62ea2956a6c53bd3ee042ce1bbed": OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/bash": stat /bin/bash: no such file or directory
So time to use an ephemeral debug container with the debug option.
To have a basic ephemeral debug container we can run the following command: kubectl debug -it pod-name --image=debug-image --target=target-container.
For this case I’m not going to use any image with more tools ,just a basic busybox image with shell (even though it has already traceroute and other tools), that will make a container as capable to have access to the shared namespace process of the target container.
So the best here is to have your own debug-image prepared with tools and useful binaries, pre-loaded to the clusters or it will be loaded after the first debug containers creation, but have an image with tools you might need for debuging network and others.
This image can be kept versioned and you can improve it by adding more tools later on as well.
kubectl debug -it simple-go-service --image=busybox --target=simple-go-service
kubectl debug -it simple-go-service --image=busybox --target=simple-go-service
Targeting container "simple-go-service". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
--profile=legacy is deprecated and will be removed in the future. It is recommended to explicitly specify a profile, for example "--profile=general".
Defaulting debug container name to debugger-6lvq9.
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
/ #
We can see on the events that the ephemeral container was created:
bash
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulling 64s kubelet spec.ephemeralContainers{debugger-6lvq9}: Pulling image "busybox"
Normal Pulled 63s kubelet spec.ephemeralContainers{debugger-6lvq9}: Successfully pulled image "busybox" in 762ms (762ms including waiting). Image size: 1911327 bytes.
Normal Created 63s kubelet spec.ephemeralContainers{debugger-6lvq9}: Container created
Normal Started 63s kubelet spec.ephemeralContainers{debugger-6lvq9}: Container started
So right now we are able to inspect what’s happening with the container we are attached, so the process namespace of the other container is shared with the ephemeral container here.
/ # whoami
root
/ # ps -ef | grep service
1 root 0:00 {service} /usr/bin/qemu-x86_64-static /bin/service
42 root 0:00 grep service
/ #
If we had logs to inspect from this app, if we wanted to see hwo the app would like survive a kill signal for example, and see how it recovers, we could all test through the debug without compromising the slim image, so without adding extra libraries and deps on the image.
A few things about ephemeral containers:
- After their creation they can’t be remove, just like regular containers, but they won’t restart in case of failures or errors.
- They can’t have any ports, no liveness / readiness config allowed as well.
- You can’t set resources to these containers.
- You can’t add ephemeral containers to static pods in the cluster, so only to your regular registered pods.
References:
Ps: I’ll be adding more debug cases and more debug options here for nodes for example later on, but I’ll have to take a read on the docs and prepare an example.