Kubernetes invisible Pod

I’ve just revisited a core concept on k8s, but through a different perspective using a great lab at the iximiuz labs platform (highly recommend the platform!). The lab is about creating an invisible pod on a kubernetes cluster using the static pod concept, and making it to not to turn into mirror pods, so not properly registered within kubernetes / etcd.

Static pods are created / managed by kubelet process on the node, so on your initial setup create those pods / resources, and their specification comes from files below the /etc/kubernetes/manifests.

Mirror pods are static pods where the kubelet api could register within the cluster through the API, so registered on etcd and completely visible as regular app pod.

When you add or change yamls at the /etc/kubernetes/manifests directoty, the kubelet container is monitoring the path and applying those changes.

The scenario to create the invisble pod:

• Create a specification below /etc/kubernetes/manifest.
• Point that pod / resource to a specific and existing namespace.
• You can still remove the namespace within k8s, so pod is not visible anymore.
• But container within the node is still running, and run by the container engine you’re using on your cluster as all the other core static pods, becoming invisble on issuing kubectl get po -A, and / or even calling the kukubelet API.
• If you recreate the namespace, kubelet will be able again to create the mirror pod, so the internal registry within the cluster.

This situation could used to exploit and run malicious pods without detection, even though the level of control of the cluster the attacker needs to implement this is high.

Generating the invisible pod

Let’s recreate the scenario and check how we can check if we have an invisible pod running on our cluster.

I’m doing this test using minikube on podman.

I’ve create my cluster with 2 nodes with the following:

minikube start --container-runtime=containerd --cni=cilium --nodes=2 --memory=1.9G --driver=podman

PS: Minimal resources for minikube on podman per node is of ~1.8GB.

Let’s create an intial namespace just to see the static pod getting registered and generating the mirror pod:

kubectl create ns invisible

Let me generate now the spec of the pod, it will be just a pod running ubuntu with a loop and sleep.

kubectl -n invisible run invisible --image=ubuntu --dry-run=client -o yaml -- bash -c "while true; do echo \"\$(date '+%Y-%m-%dT%H:%M:%S') logging my app.\"; sleep 20; done"

And here’e the yaml generated:

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: invisible
  name: invisible
  namespace: invisible
spec:
  containers:
  - args:
    - bash
    - -c
    - while true; do echo "$(date '+%Y-%m-%dT%H:%M:%S') logging my app."; sleep 20;
      done
    image: ubuntu
    name: invisible
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

Let’s go into one of the nodes and create a new manifest file with the spec above.

kubectl get nodes -o wide
NAME           STATUS   ROLES           AGE    VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION             CONTAINER-RUNTIME
minikube       Ready    control-plane   6d6h   v1.35.1   192.168.49.2   <none>        Debian GNU/Linux 12 (bookworm)   6.12.13-200.fc41.aarch64   containerd://2.2.1
minikube-m02   Ready    <none>          6d6h   v1.35.1   192.168.49.3   <none>        Debian GNU/Linux 12 (bookworm)   6.12.13-200.fc41.aarch64   containerd://2.2.1

I’ll create the manifest file at the minikube node.

Checking the containers on podman and getting into the interactive terminal:

podman ps
CONTAINER ID  IMAGE                                COMMAND     CREATED     STATUS         PORTS                                                                                                                                 NAMES
3aabe3074548  gcr.io/k8s-minikube/kicbase:v0.0.50              6 days ago  Up 47 minutes  127.0.0.1:34653->22/tcp, 127.0.0.1:34969->2376/tcp, 127.0.0.1:43805->5000/tcp, 127.0.0.1:42649->8443/tcp, 127.0.0.1:35845->32443/tcp  minikube
89c5bc7ef99c  gcr.io/k8s-minikube/kicbase:v0.0.50              6 days ago  Up 47 minutes  127.0.0.1:37373->22/tcp, 127.0.0.1:44849->2376/tcp, 127.0.0.1:44893->5000/tcp, 127.0.0.1:34337->8443/tcp, 127.0.0.1:33685->32443/tcp  minikube-m02

podman exec -ti minikube bash

We can see that we have already other manifest files at the same location, and those are for core k8s components, that follow the same concept described above about static pods / resources.

root@minikube:/# ls -ltr /etc/kubernetes/manifests/
total 24
-rw-------. 1 root root 2655 Apr  2 12:56 etcd.yaml
-rw-------. 1 root root 1727 Apr  2 12:56 kube-scheduler.yaml
-rw-------. 1 root root 3483 Apr  2 12:56 kube-controller-manager.yaml
-rw-------. 1 root root 4171 Apr  2 12:56 kube-apiserver.yaml

And on the same path I’ve just create the new manifest file with the content above:

root@minikube:/#
root@minikube:/# ls -ltr /etc/kubernetes/manifests/invisible.yaml
-rw-r--r--. 1 root root 367 Apr  8 19:14 /etc/kubernetes/manifests/invisible.yaml
root@minikube:/#
root@minikube:/#
root@minikube:/# cat /etc/kubernetes/manifests/invisible.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: invisible
    ....

We can check all static pods / resources running on containerd directly within the node as well:

crictl ps

crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                      ATTEMPT             POD ID              POD                                NAMESPACE
b4fbde18cb4b9       b933e0ff0a239       2 minutes ago       Running             invisible                 1                   74300e85310b4       invisible-minikube                 invisible
990f8dc771f4f       e08f4d9d2e6ed       52 minutes ago      Running             coredns                   1                   2fcfa78602322       coredns-7d764666f9-hvt4c           kube-system
ed57489e6b976       6aa9be0a1bb61       52 minutes ago      Running             registry-proxy            1                   c9dc6a14b5687       registry-proxy-kftff               kube-system
599667072e137       5d91beb063a96       52 minutes ago      Running             trivy-webhook             1                   458bff93a31ed       trivy-webhook-7dc78c4567-c2z2g     default
08f56108c1918       2f5ec5015badd       52 minutes ago      Running             registry                  1                   7b349aac54cbd       registry-7984dc5f99-fhsl4          kube-system
11fca1d898ffb       ba04bb24b9575       52 minutes ago      Running             storage-provisioner       2                   b42983b17115c       storage-provisioner                kube-system
c201b9851144a       6a7b645c8d911       52 minutes ago      Running             cilium-agent              1                   d37fdea215655       cilium-m6z8p                       kube-system
cd196199212f8       ae63e266f5b88       53 minutes ago      Running             cilium-envoy              1                   dc57dc4eb6fc1       cilium-envoy-7mlk4                 kube-system
338b82609ef21       b77c3cf681abd       53 minutes ago      Running             cilium-operator           1                   5885f7bded9f6       cilium-operator-8dd6995d5-jhc4t    kube-system
55ba17b03d1cf       2d611d040292f       53 minutes ago      Running             kube-proxy                1                   1c4a1e6bb9691       kube-proxy-qcwb2                   kube-system
57ecfbea0c67e       7459ea0017638       53 minutes ago      Running             kube-apiserver            1                   88ce85471b044       kube-apiserver-minikube            kube-system
743b0c40f3da2       4cad4f996631b       53 minutes ago      Running             kube-scheduler            1                   f4b6ef5692b8e       kube-scheduler-minikube            kube-system
0443fb134f8fb       271e49a0ebc56       53 minutes ago      Running             etcd                      1                   bf2b110e627e3       etcd-minikube                      kube-system
d6c1ab808351b       464c974e702ea       53 minutes ago      Running             kube-controller-manager   1                   ffda53ea21706       kube-controller-manager-minikube   kube-system

On the first line we can see our container running as part of the containerd, together with other core components of k8s.

crictl logs -f b4fbde18cb4b9

crictl logs -f b4fbde18cb4b9
2026-04-08T19:15:17 logging my app.
2026-04-08T19:15:37 logging my app.
2026-04-08T19:15:57 logging my app.
2026-04-08T19:16:17 logging my app.
2026-04-08T19:16:37 logging my app.
2026-04-08T19:16:57 logging my app.
2026-04-08T19:17:17 logging my app.
2026-04-08T19:17:37 logging my app.
2026-04-08T19:17:57 logging my app.
2026-04-08T19:18:17 logging my app.
2026-04-08T19:18:37 logging my app.
...

Ps: crictl is a universal CRI comand line for interacting with container engines, in our case containerd, and it has docker / podman compatible syntax.

Let’s check again the pods registered, or if this pod is now registered as a mirror:

k -n invisible get po
NAME                 READY   STATUS    RESTARTS   AGE
invisible-minikube   1/1     Running   1          2m9s

We can see our pod registered as a mirror pod too, and present only on our minikube node.

Let’s now remove the namespace and check internally in the node if our pod is still running or not.

k delete ns invisible

Now we see no sign of the namespace and no sign of our pod as well:

k get ns
NAME              STATUS   AGE
apps              Active   6d6h
cilium-secrets    Active   6d6h
default           Active   6d6h
kube-node-lease   Active   6d6h
kube-public       Active   6d6h
kube-system       Active   6d6h

k get po 
NAMESPACE     NAME                               READY   STATUS    RESTARTS      AGE
default       trivy-webhook-7dc78c4567-c2z2g     1/1     Running   1 (60m ago)   5d11h
default       trivy-webhook-7dc78c4567-clhlv     1/1     Running   1 (60m ago)   5d11h
kube-system   cilium-envoy-7mlk4                 1/1     Running   1 (60m ago)   6d6h
kube-system   cilium-envoy-b4mdf                 1/1     Running   1 (60m ago)   6d6h
kube-system   cilium-m6z8p                       1/1     Running   1 (60m ago)   6d6h
kube-system   cilium-operator-8dd6995d5-jhc4t    1/1     Running   1 (60m ago)   6d6h
kube-system   cilium-v552z                       1/1     Running   1 (60m ago)   6d6h
kube-system   coredns-7d764666f9-hvt4c           1/1     Running   1 (60m ago)   6d6h
kube-system   etcd-minikube                      1/1     Running   1 (60m ago)   6d6h
kube-system   kube-apiserver-minikube            1/1     Running   1 (60m ago)   6d6h
kube-system   kube-controller-manager-minikube   1/1     Running   1 (60m ago)   6d6h
kube-system   kube-proxy-qcwb2                   1/1     Running   1 (60m ago)   6d6h
kube-system   kube-proxy-xkzln                   1/1     Running   1 (60m ago)   6d6h
kube-system   kube-scheduler-minikube            1/1     Running   1 (60m ago)   6d6h
kube-system   registry-7984dc5f99-fhsl4          1/1     Running   1 (60m ago)   6d6h
kube-system   registry-proxy-kftff               1/1     Running   1 (60m ago)   6d6h
kube-system   registry-proxy-t9lg2               1/1     Running   1 (60m ago)   6d6h
kube-system   storage-provisioner                1/1     Running   2 (59m ago)   6d6h

Kubelet will keep on monitoring the manifests, and apply changes, but on this case as there’s no namespace, it’s not able to register the mirror pod.

But at the same time I could not see any logs about the pod, so it’s really hard to monitor.

Let’s see internally on the node if our container is still running.

podman exec -ti minikube bash

crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                      ATTEMPT             POD ID              POD                                NAMESPACE
b4fbde18cb4b9       b933e0ff0a239       15 minutes ago      Running             invisible                 1                   74300e85310b4       invisible-minikube                 invisible
990f8dc771f4f       e08f4d9d2e6ed       About an hour ago   Running             coredns                   1                   2fcfa78602322       coredns-7d764666f9-hvt4c           kube-system
ed57489e6b976       6aa9be0a1bb61       About an hour ago   Running             registry-proxy            1                   c9dc6a14b5687       registry-proxy-kftff               kube-system
599667072e137       5d91beb063a96       About an hour ago   Running             trivy-webhook             1                   458bff93a31ed       trivy-webhook-7dc78c4567-c2z2g     default
08f56108c1918       2f5ec5015badd       About an hour ago   Running             registry                  1                   7b349aac54cbd       registry-7984dc5f99-fhsl4          kube-system
11fca1d898ffb       ba04bb24b9575       About an hour ago   Running             storage-provisioner       2                   b42983b17115c       storage-provisioner                kube-system
c201b9851144a       6a7b645c8d911       About an hour ago   Running             cilium-agent              1                   d37fdea215655       cilium-m6z8p                       kube-system
cd196199212f8       ae63e266f5b88       About an hour ago   Running             cilium-envoy              1                   dc57dc4eb6fc1       cilium-envoy-7mlk4                 kube-system
338b82609ef21       b77c3cf681abd       About an hour ago   Running             cilium-operator           1                   5885f7bded9f6       cilium-operator-8dd6995d5-jhc4t    kube-system
55ba17b03d1cf       2d611d040292f       About an hour ago   Running             kube-proxy                1                   1c4a1e6bb9691       kube-proxy-qcwb2                   kube-system
57ecfbea0c67e       7459ea0017638       About an hour ago   Running             kube-apiserver            1                   88ce85471b044       kube-apiserver-minikube            kube-system
743b0c40f3da2       4cad4f996631b       About an hour ago   Running             kube-scheduler            1                   f4b6ef5692b8e       kube-scheduler-minikube            kube-system
0443fb134f8fb       271e49a0ebc56       About an hour ago   Running             etcd                      1                   bf2b110e627e3       etcd-minikube                      kube-system
d6c1ab808351b       464c974e702ea       About an hour ago   Running             kube-controller-manager   1                   ffda53ea21706       kube-controller-manager-minikube   kube-system
root@minikube:/#

And we can still see the container running within the node.

crictl logs -f b4fbde18cb4b9

crictl logs -f b4fbde18cb4b9
2026-04-08T19:15:17 logging my app.
2026-04-08T19:15:37 logging my app.
2026-04-08T19:15:57 logging my app.
2026-04-08T19:16:17 logging my app.
2026-04-08T19:16:37 logging my app.
2026-04-08T19:16:57 logging my app.
2026-04-08T19:17:17 logging my app.
2026-04-08T19:17:37 logging my app.
2026-04-08T19:17:57 logging my app.
2026-04-08T19:18:17 logging my app.
2026-04-08T19:18:37 logging my app.
2026-04-08T19:18:57 logging my app.
2026-04-08T19:19:17 logging my app.
2026-04-08T19:19:37 logging my app.
2026-04-08T19:19:57 logging my app.
2026-04-08T19:20:17 logging my app.
2026-04-08T19:20:37 logging my app.
2026-04-08T19:20:57 logging my app.
2026-04-08T19:21:17 logging my app.
2026-04-08T19:21:37 logging my app.
2026-04-08T19:21:57 logging my app.
2026-04-08T19:22:17 logging my app.
2026-04-08T19:22:37 logging my app.
2026-04-08T19:22:57 logging my app.
2026-04-08T19:23:17 logging my app.
2026-04-08T19:23:37 logging my app.
2026-04-08T19:23:57 logging my app.
2026-04-08T19:24:17 logging my app.
2026-04-08T19:24:37 logging my app.
2026-04-08T19:24:57 logging my app.
2026-04-08T19:25:17 logging my app.
2026-04-08T19:25:37 logging my app.
2026-04-08T19:25:57 logging my app.
2026-04-08T19:26:17 logging my app.
2026-04-08T19:26:37 logging my app.
2026-04-08T19:26:57 logging my app.
2026-04-08T19:27:17 logging my app.
2026-04-08T19:27:37 logging my app.
2026-04-08T19:27:57 logging my app.
2026-04-08T19:28:17 logging my app.
2026-04-08T19:28:37 logging my app.
2026-04-08T19:28:57 logging my app.
2026-04-08T19:29:17 logging my app.
2026-04-08T19:29:37 logging my app.
2026-04-08T19:29:57 logging my app.
2026-04-08T19:30:17 logging my app.
2026-04-08T19:30:37 logging my app.
2026-04-08T19:30:57 logging my app.
2026-04-08T19:31:17 logging my app.
2026-04-08T19:31:37 logging my app.
2026-04-08T19:31:57 logging my app.
2026-04-08T19:32:17 logging my app.
2026-04-08T19:32:37 logging my app.
...

So container is still running, and we can’t actually see the pod within k8s using kubectl.

k get po -A
NAMESPACE     NAME                               READY   STATUS    RESTARTS      AGE
default       trivy-webhook-7dc78c4567-c2z2g     1/1     Running   1 (70m ago)   5d11h
default       trivy-webhook-7dc78c4567-clhlv     1/1     Running   1 (70m ago)   5d11h
kube-system   cilium-envoy-7mlk4                 1/1     Running   1 (70m ago)   6d6h
kube-system   cilium-envoy-b4mdf                 1/1     Running   1 (70m ago)   6d6h
kube-system   cilium-m6z8p                       1/1     Running   1 (70m ago)   6d6h
kube-system   cilium-operator-8dd6995d5-jhc4t    1/1     Running   1 (70m ago)   6d6h
kube-system   cilium-v552z                       1/1     Running   1 (70m ago)   6d6h
kube-system   coredns-7d764666f9-hvt4c           1/1     Running   1 (70m ago)   6d6h
kube-system   etcd-minikube                      1/1     Running   1 (70m ago)   6d6h
kube-system   kube-apiserver-minikube            1/1     Running   1 (70m ago)   6d6h
kube-system   kube-controller-manager-minikube   1/1     Running   1 (70m ago)   6d6h
kube-system   kube-proxy-qcwb2                   1/1     Running   1 (70m ago)   6d6h
kube-system   kube-proxy-xkzln                   1/1     Running   1 (70m ago)   6d6h
kube-system   kube-scheduler-minikube            1/1     Running   1 (70m ago)   6d6h
kube-system   registry-7984dc5f99-fhsl4          1/1     Running   1 (70m ago)   6d6h
kube-system   registry-proxy-kftff               1/1     Running   1 (70m ago)   6d6h
kube-system   registry-proxy-t9lg2               1/1     Running   1 (70m ago)   6d6h
kube-system   storage-provisioner                1/1     Running   2 (69m ago)   6d6h

So if an attacker can get and inject something on your manifest directory, and generate a new pod with a non-existent namespace, the pod can start and keep on running.

That shows how important is to lock up access directly to nodes / cluster, and audit the activity within it.

In this case the attacker has to have access to generate a static pod, so access to the nodes directly, and one way to have it secure on GCP for example is to have private clusters, so no external IPs, firewall rules, and controlling even egreess with NAT or just allowing intenal traffic / GCP VPC / subnet.

How to best monitor or check for invisible pods

• Telemetry within the node - shows what’s running in terms of internal containers within containerd.
  • No errors are spawned from the kubelet, so hard to actually find the issue.
• kubeletctl - project is not being maintained anymore and I had errors with current k8s versions.
• Query directly the kubelet API - shows all pods including the static / invisible ones.
  • You have to get all certs to do the curl call directly to the api:

      kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt
      kubectl config view --raw --minify --flatten -o jsonpath='{.users[0].user.client-certificate-data}' | base64 -d > client.crt
      kubectl config view --raw --minify --flatten -o jsonpath='{.users[0].user.client-key-data}' | base64 -d > client.key
    

  • Use the certs now to do your curl call, but anyway there’s no list of the pod, only an error from it (Ps: change your ip / port if needed, on my case it’s minikube on podman):

      curl -v --cacert ca.crt --cert client.crt --key client.key https://127.0.0.1:42649/api/v1/pods
    

  • In my case I could find some small errors for a different static pod, but it showed nothing on the invisble one.
• In summary:
  • Have an audit methodology that can check what’s running on your nodes, not just internally registered in the cluster.
  • Run most nodes on cloud platforms on private mode, no exposing any IP nodes externally.
  • Restricting access to the kubelet APIs, and access to the nodes directly is imperative to avoid the case above.
  • Check resources getting created with any securityContext and privileged requirements, as on those are the ones that could be even generating something on the fly.
  • Audit container images, as compromise images running on high privileged pods / resources as explained above.

References:

• iximiuz labs platform
• iximiuz lab invisible pod
• Static pods from k8s docs
• Medium page from Anshuman Tripathi with a similar test with kind