Debugging Containers and Clusters

Everyone's first Kubernetes day ends the same way: you apply a file, run kubectl get pods, and see a status you do not understand. The good news is that the number of statuses is small, and each has a short list of causes. The skill is to avoid guessing, and instead to read the one place that holds the answer.

This lesson teaches that habit. First the Docker side, where I can run every example for real. Then the Kubernetes side, where I show real output wherever my lab can produce it, and I say clearly where I am using the documented text of a failure instead. At the end there is a decision table you can keep.

This is lesson 8 of the series (lessons run from 0 to 9). It uses the ideas from Scaling, Rollouts and Reliability (probes, limits, OOM kills) and Configuration, Secrets and Storage (a missing ConfigMap or a claim that cannot bind). The next lesson, Docker and Kubernetes Interview Questions With Model Answers, turns all of this into answers. If Docker itself is new, start with Why Containers.

Words used in this lesson

• Exit code is the number a process returns when it ends. 0 means success. Anything else means some kind of failure.
• Signal is a message the kernel sends to a process, such as SIGTERM (please stop) or SIGKILL (stop now, no choice).
• PID 1 is the first process in a container. Linux treats it specially: it ignores signals it has no handler for.
• Event is a short record Kubernetes writes when something happens to an object ("Scheduled", "FailedScheduling", "Pulled").
• Status (in kubectl get pods) is a display word that kubectl builds from the pod. It is not an official state.
• Phase is the official one-word state of a pod: Pending, Running, Succeeded, Failed or Unknown.
• CrashLoopBackOff means a container keeps crashing and Kubernetes waits longer and longer between restarts.
• ImagePullBackOff means the node cannot download the image and keeps retrying with a growing delay.
• OOMKilled means the kernel stopped the container for using more memory than its limit. The exit code is 137.
• Ephemeral container is a temporary extra container that you add to a running pod just to debug it.
• Port-forward is a tunnel from a port on your laptop to a port on a pod.
• Endpoint is one address behind a Service. An empty list means the Service has nowhere to send traffic.
• Node is a machine in the cluster. NotReady means its kubelet has stopped reporting that it is healthy.

A method that works

Before any command, here is the loop I follow every time. It is the same for Docker and Kubernetes.

1. State. What is the thing doing right now? Running, exited, restarting, pending? How many restarts?
2. Logs. What did the program itself say before it died or while it misbehaves?
3. Events or exit code. What did the platform say about it? (Kubernetes events, Docker exit code and OOMKilled flag.)
4. Compare. What is different from the version that worked? A new image, a new setting, a new limit?
5. Change one thing, and check again.

Most people jump to step 5. They edit the YAML and re-apply, hoping. Steps 1 to 3 cost seconds and nearly always name the cause.

Part 1: Docker, with real runs

A container that exits at once

Here is a small Node.js program that needs a setting and exits if it is missing. This is the same mistake as forgetting a ConfigMap key in lesson 6.

// crash.jsconst need = process.env.DATABASE_URL;console.log("starting notes-api");if (!need) {  console.error("FATAL: DATABASE_URL is not set");  process.exit(1);}console.log("connected to", need);

Run it in the background and look:

docker run -d --name l8-crash -v $PWD/crash.js:/app/crash.js:ro \  mirror.gcr.io/library/node:22-alpine node /app/crash.jsdocker ps -a --filter name=l8-crash --format 'table {{.Names}}\t{{.Status}}'docker logs l8-crashdocker inspect l8-crash --format 'exit={{.State.ExitCode}} oom={{.State.OOMKilled}} status={{.State.Status}}'

NAMES      STATUSl8-crash   Exited (1) 1 second agostarting notes-apiFATAL: DATABASE_URL is not setexit=1 oom=false status=exited

Three lessons are in this small output.

• docker ps shows only running containers. docker ps -a (all) shows the exited ones. A beginner who runs plain docker ps sees nothing and thinks the container "disappeared".
• docker logs shows what the program wrote to stdout and stderr, even after it died. The reason is right there.
• docker inspect gives you the exact exit code and the OOMKilled flag. Exit code 1 is the generic "the program failed".

The fix is to pass the setting. Running again with -e DATABASE_URL=postgres://db/notes prints connected to postgres://db/notes and exits normally.

Exit codes, and what they usually mean

I collected these from real runs in my lab. Codes 125, 126 and 127 are returned by the docker command itself.

The rule of thumb: an exit code above 128 means "killed by a signal", and the signal number is the code minus 128. That gives you 137 = 9 (KILL) and 143 = 15 (TERM). Kubernetes shows the same numbers.

The restart loop

A container with a restart policy shows what a crash loop looks like. I ran the same crashing program with --restart on-failure:

docker run -d --name l8-loop --restart on-failure \  -v $PWD/crash.js:/app/crash.js:ro mirror.gcr.io/library/node:22-alpine node /app/crash.jssleep 6docker ps -a --filter name=l8-loop --format '{{.Names}} | {{.Status}}'docker inspect l8-loop --format 'restarts={{.RestartCount}} status={{.State.Status}} exit={{.State.ExitCode}}'docker logs l8-loop | sort | uniq -c

l8-loop | Restarting (1) 1 second agorestarts=6 status=restarting exit=1      6 FATAL: DATABASE_URL is not set      6 starting notes-api

In six seconds the container started six times. The RestartCount is the number to watch, and the logs show the same error each time. This is what Kubernetes' CrashLoopBackOff is: the same loop, but with a growing wait between tries so it does not burn the node. Docker's own restart policy also backs off (its delay starts small and doubles), which is why a long-running loop looks slower than my six-second sample.

OOMKilled, for real

Here is a program that holds on to more and more memory:

// hog.jsconst chunks = [];console.log("allocating memory");setInterval(() => { chunks.push(Buffer.alloc(10 * 1024 * 1024, 1)); console.log("held", chunks.length * 10, "MiB"); }, 100);

I ran it with a 64 MiB limit and no swap:

docker run -d --name l8-oom --memory 64m --memory-swap 64m \  -v $PWD/hog.js:/app/hog.js:ro mirror.gcr.io/library/node:22-alpine node /app/hog.jssleep 4docker ps -a --filter name=l8-oom --format '{{.Names}} | {{.Status}}'docker inspect l8-oom --format 'exit={{.State.ExitCode}} oom={{.State.OOMKilled}}'docker logs l8-oom | tail -3

l8-oom | Exited (137) 3 seconds agoexit=137 oom=trueheld 30 MiBheld 40 MiBheld 50 MiB

This is the whole story of an OOM kill. The program printed held 50 MiB, tried to grow past the limit, and the kernel killed it. There is no error message in the logs. The program did not get a chance to say anything, because it was killed by SIGKILL. The only clues are the exit code 137 and the flag oom=true. This is why "no error in the logs, but the pod keeps restarting" should make you think of memory first.

Looking inside a running container

For a container that is running, these are the tools. I ran each against a small Node HTTP server (server.js, which answers hello from notes-api and has a /healthz path):

docker run -d --name l8-web -p 18080:3000 -v $PWD/server.js:/app/server.js:ro \  mirror.gcr.io/library/node:22-alpine node /app/server.jsdocker logs l8-webdocker ps --filter name=l8-web --format '{{.Names}} | {{.Status}} | {{.Ports}}'docker port l8-webcurl -s localhost:18080/healthz

listening on 3000l8-web | Up 1 second | 0.0.0.0:18080->3000/tcp3000/tcp -> 0.0.0.0:18080ok

Now exec, top, stats, and diff:

docker exec l8-web sh -c 'id; ls /app; wget -qO- localhost:3000/healthz'docker top l8-webdocker stats --no-stream --format 'name={{.Name}} cpu={{.CPUPerc}} mem={{.MemUsage}} pids={{.PIDs}}' l8-webdocker exec l8-web sh -c 'echo x > /tmp/scratch.txt'docker diff l8-web

uid=0(root) gid=0(root) groups=0(root),0(root),1(bin),2(daemon),...server.jsokUID   PID     PPID    C   STIME   TTY   TIME       CMDroot  24258   24232   5   03:13   ?     00:00:00   node /app/server.jsname=l8-web cpu=0.03% mem=9.352MiB / 15.72GiB pids=7A /appA /app/server.jsC /tmpA /tmp/scratch.txt

What each one tells you:

• exec runs a command inside the container. First, look at id: it says uid=0(root). The container runs as root, which lesson 2 told you to avoid.
• top shows the processes of this container as the host sees them. Only one: node /app/server.js.
• stats shows live CPU and memory. Memory 9.3MiB and mem=... / 15.72GiB: with no limit, the "limit" shown is the whole machine.
• diff lists files that changed in the container's writable layer compared to the image (A added, C changed, D deleted). It is great for "where is my app writing files?".

Debugging the network with a sidecar

When the app image has no tools, you can start a second container that shares the first container's network. Then localhost means the same thing in both:

docker run --rm --network container:l8-web mirror.gcr.io/library/busybox:1.36 wget -qO- localhost:3000/

hello from notes-api

This is the same idea as a Kubernetes sidecar and an ephemeral debug container: put your tools in a separate container, and share the namespace of the broken one.

Signals and docker stop: a trap with a real number

docker stop sends SIGTERM and, after 10 seconds, SIGKILL. A well-behaved server stops quickly. I tested two servers. One has a SIGTERM handler:

process.on("SIGTERM", () => { console.log("got SIGTERM, bye"); srv.close(() => process.exit(0)); });

$ time docker stop l8-webreal    0m0.299s$ docker logs l8-web | tail -2listening on 3000got SIGTERM, byeexit=0

The other server (server-nosig.js) has no handler. Because it runs as PID 1 in the container, and Linux does not apply the default "terminate" action for signals to PID 1 unless the program installs a handler, the SIGTERM is simply ignored:

$ time docker stop l8-nosigreal    0m10.216sexit=137

It took the whole 10 seconds and ended with 137 (killed). Now the same server with --init, which puts a tiny init program (tini) in front as PID 1 to forward signals and clean up zombie processes:

$ time docker stop l8-initreal    0m0.208sexit=143

Quick and clean (143 is SIGTERM). In Kubernetes the same trap makes every pod take the full terminationGracePeriodSeconds (30 s) to stop, which makes rollouts slow. It also happens when a Dockerfile uses the shell form CMD node server.js, because then /bin/sh is PID 1 and may not pass the signal on. Use the exec form CMD ["node", "server.js"] (see lesson 1).

Start-up failures with the exact error text

All of these are real Docker errors from my runs. Learn to recognise them.

Command not found:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "nope": executable file not found in $PATH

The long prefix is Docker's plumbing. The useful part is at the end: exec: "nope": executable file not found in $PATH. Typo in CMD, or the binary is not in the image.

File not found (the app, not the runtime):

Error: Cannot find module '/app/missing.js'  code: 'MODULE_NOT_FOUND',

Here Node started fine and then could not find the file. The error comes from your program. Check WORKDIR, COPY paths and .dockerignore.

Image tag does not exist:

Error response from daemon: failed to resolve reference "mirror.gcr.io/library/node:22-doesnotexist": mirror.gcr.io/library/node:22-doesnotexist: not found

Port already in use:

docker: Error response from daemon: failed to set up container networking: driver failed programming external connectivity on endpoint l8-b (ffd0774015bc...): Bind for 0.0.0.0:18081 failed: port is already allocated

Note what docker ps -a then showed: l8-b | Created. The container was created but never started. The cause: another container already published host port 18081. Fix by choosing another host port (-p 18082:3000).

Binary built for the wrong environment. I made a tiny image FROM scratch that holds only a copy of busybox. My first try copied a dynamically linked busybox, and the container died with:

exec /busybox: no such file or directory

with exit code 255. The file exists. What is missing is the loader (the dynamic linker) that the binary needs, which a scratch image does not have. This confusing message ("no such file" for a file that is there) is a classic for people who build in one base image and run in another. The fix was a static busybox (the 1.36-musl variant).

An image with no shell

That static scratch image is a good test for a very common problem: a minimal or "distroless" image has no sh, so docker exec -it ... sh fails.

$ docker exec l8-nosh shOCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH

You are not stuck. Attach a tools container that shares the namespaces of the broken one:

docker run --rm --network container:l8-nosh mirror.gcr.io/library/busybox:1.36 wget -qO- localhost:8080docker run --rm --pid container:l8-nosh mirror.gcr.io/library/busybox:1.36 ps

static pagePID   USER     TIME  COMMAND    1 1000      0:00 /busybox httpd -f -p 8080 -h /www   13 root      0:00 ps

The first command reached the app through its network. The second sees its processes (the web server is PID 1, running as user 1000). This is how I would debug a distroless image on a laptop. In Kubernetes there is a ready-made version of this: the ephemeral container, below. (When I tried to read the other container's files through /proc/1/root, I got Permission denied, because the server ran as user 1000 and my tools container ran as root without the right capabilities. Namespaces are a boundary, and sharing a PID namespace does not give you free access to everything.)

Part 2: Kubernetes

The first four commands

When a pod looks wrong, I run these in order.

kubectl get pods -o wide        # state, restarts, which node, which IPkubectl describe pod NAME       # everything about it, with EVENTS at the bottomkubectl logs NAME               # what the app printedkubectl get events --sort-by=.lastTimestamp   # recent events in the namespace

kubectl describe is the most useful of the four. Its last section, Events, is a short diary of what the scheduler and the kubelet did and what went wrong. Here is a real one, for a pod that cannot be scheduled. I made a pod that asks for 100 CPUs:

$ kubectl -n lab8 describe pod too-bigName:         too-bigNamespace:    lab8Node:         <none>Status:       PendingContainers:  app:    Image:      notes-api:1.0    Requests:      cpu:        100      memory:     1GiConditions:  Type           Status  PodScheduled   FalseQoS Class:       BurstableEvents:  Type     Reason            Age    From               Message  ----     ------            ----   ----               -------  Warning  FailedScheduling  8m37s  default-scheduler  0/1 nodes are available: 1 Insufficient cpu. no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

(I removed some lines for space.) See how much it tells you. Node: <none> and PodScheduled False say the scheduler has not placed it. The event says why: Insufficient cpu.

kubectl logs: the flags you need

kubectl logs NAME                      # the one container (if the pod has one)kubectl logs NAME -c CONTAINER         # pick a container in a multi-container podkubectl logs NAME --previous           # the logs of the LAST crashed instancekubectl logs -f NAME                   # follow, like tail -fkubectl logs NAME --since=10m          # only recent lineskubectl logs -l app=notes-api --tail=20  # by label, last 20 lines of each

The one to remember is --previous. When a container crashes and restarts, plain kubectl logs shows the new instance, which may have just started and printed nothing yet. The reason for the crash is in the previous instance. The Kubernetes docs list --previous for exactly this.

kubectl exec, port-forward and debug

kubectl exec -it NAME -- sh            # a shell in the containerkubectl exec NAME -- cat /etc/notes/app.json   # one commandkubectl port-forward pod/NAME 8080:3000   # laptop:8080 -> pod:3000kubectl port-forward svc/NAME 8080:80     # same, through a Service

When the pod is not running, these fail early with a clear message. This one is real:

$ kubectl -n lab8 port-forward pod/web-89f547c8c-8bs4b 8080:3000error: unable to forward port because pod is not running. Current status=Pending

Ephemeral containers are the answer to the "no shell in the image" problem from the Docker half. kubectl debug adds a temporary container (with tools such as busybox) into a running pod, and --target makes it share the process namespace of your app container. I ran it against a pod in my lab. The API server accepted it, and the pod now had this in its spec (real output, shown as JSON):

[  {    "command": ["sleep", "3600"],    "image": "mirror.gcr.io/library/busybox:1.36",    "imagePullPolicy": "IfNotPresent",    "name": "dbg",    "resources": {},    "securityContext": { "capabilities": { "add": ["SYS_PTRACE"] } },    "targetContainerName": "notes-api",    "terminationMessagePath": "/dev/termination-log",    "terminationMessagePolicy": "File"  }]

Notice targetContainerName (which container's namespaces to share) and that the general profile added the SYS_PTRACE capability so tools like strace can attach. In real use you add -it and get a shell: kubectl debug -it NAME --image=busybox:1.36 --target=CONTAINER. Mistyping --target is caught by the server, as I found out. My first try used the wrong container name, and the answer was spec.ephemeralContainers[0].targetContainerName: Not found: "web". (The container name came from the image name, notes-api, since I had made the Deployment with kubectl create deployment.) Two limits: an ephemeral container cannot be removed once added (delete the pod to get rid of it), and it only works on a pod that has been scheduled.

Two more kubectl debug forms, from the docs:

• --copy-to=NAME2 makes a copy of the pod with a change (a different image, or a different command). It is for pods that crash at start, where you cannot exec because the container never stays up. For example: copy the pod, change the command to sleep 3600, and exec into the copy to look around. I ran the copy form against my lab and a web-debug pod appeared next to the original.
• kubectl debug node/NAME -it --image=ubuntu starts a pod on that node with the node's filesystem mounted at /host. It is for node-level problems.

The status words, and where the answer is

Here is the table I use. The "meaning" column is from the Kubernetes docs (Pod lifecycle, Images, resource management). I could not produce these states in my lab, because they need a kubelet, so I do not show terminal output for them. From the docs, not run on a cluster here.

Notice the pattern: the answer is almost always in describe events, logs --previous, or the exit code.

Pending: three real causes, three real messages

Pending is the one status I can fully show, because the scheduler is real. I made three pods that could never be placed, each for a different reason. These are real events from my lab.

1. Not enough CPU or memory (the pod asked for 100 CPUs):

Warning  FailedScheduling  default-scheduler  0/1 nodes are available: 1 Insufficient cpu. no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

2. The node selector matches no node (the pod wants a node labelled disktype: ssd):

Warning  FailedScheduling  default-scheduler  0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

3. The volume claim does not exist (the pod mounts a claim named no-such-claim):

Warning  FailedScheduling  default-scheduler  0/1 nodes are available: persistentvolumeclaim "no-such-claim" not found. not found

The format is always the same: 0/N nodes are available: followed by the count of nodes and the reason for each group. Read the reasons. They are the whole answer. Other reasons you will see: node(s) had untolerated taint (the node refuses pods without a matching toleration; I saw {node.kubernetes.io/unreachable: } in my lab when the fake node's heartbeat stopped),.

Other Pending causes from the earlier lessons: a PVC that is itself Pending (lesson 6; real event no persistent volumes available for this claim and no storage class is set), and a ResourceQuota that stops the pod from being created at all (lesson 7; the pod does not even appear, the error is on the ReplicaSet).

One honest note: the pods crashy, typo-tag and memory-hog (their manifests are in the next sections) were scheduled (event Successfully assigned lab8/crashy to fake-node-1) but stayed Pending, because nothing could start them. In a real cluster they would move on to the states in the next sections.

CrashLoopBackOff

What it is. The container starts, exits, and the kubelet restarts it with a delay that doubles (10 s, 20 s, 40 s, and so on, capped at 5 minutes). It resets after the container has run for 10 minutes without a problem. CrashLoopBackOff is the display word for "waiting to restart".

Common causes (listed in the docs): the app has a bug or exits on bad config (our missing DATABASE_URL is exactly this, and the Docker restart loop above is the same thing), the container has too little memory or CPU, or a liveness or startup probe keeps failing.

What to run:

kubectl logs NAME --previous      # why did the last run die?kubectl describe pod NAME         # Last State: exit code, reason; events

This is a pod I wrote to show it (it prints an error and exits 1):

apiVersion: v1kind: Podmetadata:  name: crashyspec:  restartPolicy: Always  containers:    - name: app      image: mirror.gcr.io/library/busybox:1.36      command: ["sh", "-c", "echo 'FATAL: DATABASE_URL is not set' >&2; exit 1"]

Reading describe for it, per the docs: in the container section you would look at State: Waiting, Reason: CrashLoopBackOff, and Last State: Terminated with an Exit Code and a Reason. Exit code 1 and an error in --previous logs means "my code failed". Exit code 137 means "killed", so look at OOM and probes. Exit code 127 or 126 means "bad command".

ImagePullBackOff

What it is. The node tried to download the image and failed, and it will retry with growing delays (up to 5 minutes). ErrImagePull is the first failure, ImagePullBackOff is the waiting.

Common causes (from the Images docs and the debug guide): the image name or tag has a typo, the tag does not exist (we saw the Docker error not found above for node:22-doesnotexist), the image is in a private registry and the pod has no imagePullSecrets (the Secret from lesson 6), or the registry is rate limiting or unreachable from the node.

What to run: kubectl describe pod NAME, and read the Events for the Failed and Pulling reasons and the registry's message. Then test the pull yourself from a machine with the same access: docker pull IMAGE. The Kubernetes debug guide suggests exactly this. If you get not found, it is the name or tag. If you get unauthorized, it is credentials.

apiVersion: v1kind: Podmetadata:  name: typo-tagspec:  containers:    - name: app      image: mirror.gcr.io/library/node:22-doesnotexist

OOMKilled

What it is. The container used more memory than its limit, and the kernel killed it. In kubectl describe pod, under the container, the docs show Last State: Terminated with Reason: OOMKilled and Exit Code: 137, and a Restart Count that grows. Like the Docker run above, there is no message in the logs.

What to run: kubectl describe pod NAME and look at Last State. Then check memory use over time (kubectl top pod, which needs metrics-server; in my lab it printed error: Metrics API not available, the classic sign that it is missing).

Fix: find the leak or the large allocation. If the app is healthy and just needs more, raise the memory limit and the request together. Do not only raise the limit: the scheduler packs by requests, so a pod with a low request and a high limit can push a node into memory pressure.

The pod I wrote for this test (a Node program that keeps allocating, limit 64Mi):

apiVersion: v1kind: Podmetadata:  name: memory-hogspec:  containers:    - name: app      image: mirror.gcr.io/library/node:22-alpine      command: ["node", "-e", "const a=[];setInterval(()=>a.push(Buffer.alloc(1e7,1)),100)"]      resources:        requests:          memory: 32Mi        limits:          memory: 64Mi

Readiness failures and empty Services

A pod that is Running but 0/1 ready has a failing readiness probe. describe shows probe failure events. A Service in front of it then has no endpoints. This is the real Service check from lesson 5, and I ran it. I exposed a Deployment whose only pod was not ready:

$ kubectl -n lab8 get endpointslices -l kubernetes.io/service-name=webNAME        ADDRESSTYPE   PORTS     ENDPOINTS   AGEweb-lrgpc   IPv4          <unset>   <unset>     0s$ kubectl -n lab8 describe svc web | grep -E 'Selector|Endpoints'Selector:                 app=webEndpoints:

ENDPOINTS <unset> and an empty Endpoints: line mean no ready pod matches. There are two causes, and you must tell them apart: (a) the Service selector does not match the pod labels (a typo), or (b) the labels match but the pods are not ready. Check with kubectl get pods -l app=web (using the Service's selector). If pods show up there but not ready, go to readiness. If nothing shows up, the selector is wrong.

Service debugging in order

When "I cannot reach my app", walk the path in order, nearest to farthest:

1. Is the pod Running and Ready? (kubectl get pods)
2. Does the app answer inside the pod? (kubectl exec and wget/curl localhost:PORT, or kubectl port-forward pod/NAME)
3. Does the Service have endpoints? (kubectl get endpointslices -l kubernetes.io/service-name=NAME)
4. Does the Service targetPort match the port the app listens on?
5. Does DNS work from another pod? (kubectl exec OTHERPOD -- nslookup NAME)
6. Is a NetworkPolicy blocking it? (lesson 5)
7. Is the Ingress and its controller healthy? (kubectl describe ingress NAME)

Each step removes one layer. Going nearest to farthest means that when a step fails, you know the layer.

Everything is applied but "nothing happens": check the object, not your file

The Kubernetes debug guide has a point I like: if a pod runs but does not do what you meant, the YAML may have been accepted with part of it ignored, for example a misspelled key. Two checks. First, use strict validation: in my tests the typo envfrom was accepted by non-strict kubeconform and caught by kubeconform -strict and by the real API server (strict decoding error: unknown field ...), as lesson 6 showed. Second, compare what you sent with what the server stored: kubectl get pod NAME -o yaml and look for fields you wrote that are missing. And kubectl explain pod.spec.containers.resources (needs a cluster connection) prints the documentation of any field.

Cluster-level problems

If many pods on one node have trouble, look at the node.

kubectl get nodeskubectl describe node NAME

A healthy node is Ready. A node whose kubelet stopped reporting becomes NotReady, and, according to the docs, its pods are evicted after about five minutes. describe node lists Conditions (Ready, MemoryPressure, DiskPressure, PIDPressure), Taints, and Allocated resources. Here is a real piece from my lab node, which is useful for judging "is this node full?":

Allocated resources:  (Total limits may be over 100 percent, i.e., overcommitted.)  Resource           Requests     Limits  --------           --------     ------  cpu                1 (50%)      0 (0%)  memory             512Mi (12%)  1Gi (25%)

Remember from lesson 7: this is requests, not usage. Also notice the note that limits may be over 100%: limits can add up to more than the machine has (overcommit), while requests cannot.

For control-plane problems (the API server is slow, a controller does nothing), the docs point to the logs of the control-plane components and kubectl cluster-info dump. On a managed cloud cluster you cannot see those; the provider's tools do. For a self-managed node, the kubelet logs (usually via journalctl -u kubelet) say why a node is not healthy. A short list of node-level causes: the kubelet is down, the disk is full (DiskPressure), the container runtime is stuck, the network plugin is not working, or the machine is gone.

Putting it together: a flow you can follow

Commands to know

Try it

You can do the Docker steps on any machine with Docker.

1. Run docker run -d --name t1 mirror.gcr.io/library/busybox:1.36 sh -c 'echo boom >&2; exit 3' (or use busybox from Docker Hub). docker ps shows nothing. How do you find the container, the message and the exit code?
2. Run a container with --memory 32m that allocates memory in a loop (adapt hog.js). What exit code and flag do you expect? Is there an error in the logs?
3. A Node server in your image ignores SIGTERM. Give two different fixes (one in the Dockerfile or run command, one in the code).
4. A Kubernetes pod is Pending. describe shows 0/3 nodes are available: 3 node(s) had untolerated taint. What does it mean, and what are two ways to fix it?
5. A pod is in CrashLoopBackOff. kubectl logs POD prints nothing. What do you run next?
6. A Service has an empty endpoints list. Which two commands tell you if it is a selector problem or a readiness problem?
7. Your image has no shell. Name two ways to inspect a running container from Kubernetes or Docker.

Answers

1. docker ps -a lists it as Exited (3). docker logs t1 prints boom. docker inspect t1 --format '{{.State.ExitCode}}' prints 3.
2. Exit code 137 and OOMKilled true in docker inspect. There is no error in the logs because the process was killed by SIGKILL and had no chance to print.
3. In the run command or Dockerfile: use the exec form for the command and run with --init (or add tini) so signals are forwarded. In the code: add process.on("SIGTERM", ...) that closes the server and exits.
4. All three nodes have a taint that the pod does not tolerate. Fix by adding a matching tolerations entry to the pod, or by removing or changing the taint on the nodes, or by adding nodes without the taint.
5. kubectl logs POD --previous (logs of the crashed instance) and kubectl describe pod POD (exit code, reason, events).
6. kubectl get pods -l <the Service's selector>: if no pods show up, the selector or labels are wrong; if pods show up but are not READY, it is readiness (check describe pod events). Also kubectl get endpointslices -l kubernetes.io/service-name=NAME.
7. Kubernetes: kubectl debug with an ephemeral container (--target), or --copy-to a copy with a different image. Docker: run a tools container with --network container:NAME and --pid container:NAME.

Interview questions

Key takeaways

• Use the loop: state, logs, events or exit code, compare, change one thing.
• Docker: ps -a, logs, inspect (exit code, OOMKilled), exec, diff, stats. Exit codes above 128 mean "killed by signal": 137 is KILL, 143 is TERM.
• An OOM kill leaves no log message. Look for exit 137 and OOMKilled.
• A process that ignores SIGTERM as PID 1 makes docker stop wait 10 s, and a pod wait 30 s. Handle the signal, use the exec form, use --init.
• Kubernetes: describe events are the answer to Pending and image problems, and logs --previous is the answer to CrashLoopBackOff.
• Pending events have a fixed shape: 0/N nodes are available: plus reasons (Insufficient cpu, selector, taint, missing claim).
• ImagePullBackOff: check name, tag, credentials, then docker pull it yourself.
• Empty Service endpoints mean a selector typo or unready pods. Check which.
• kubectl debug gives you an ephemeral container for pods with no shell, --copy-to for pods that crash at start.
• Status is a display word. Phase is the real state.

What's next

Next: Docker and Kubernetes Interview Questions With Model Answers, Lesson 9. It is the last lesson. It collects the questions interviewers ask about containers, images, networking, orchestration and troubleshooting, with model answers and the traps. Practise now: start a container that fails, and find the cause using only docker ps -a, logs and inspect.