Scaling, Rollouts and Reliability: Probes, Resources and Autoscaling

In the last lesson your pods got settings, secrets and disks. Now comes the part that decides whether your app survives a normal Tuesday: a new version goes out, a node is drained for an upgrade, traffic doubles at lunch time, and one container has a memory leak.

Kubernetes can handle all of that, but only if you tell it the truth about your app. Four pieces of YAML carry that truth: probes, resources, update strategy and disruption budgets. Each one is small. Each one has a trap that I have seen in real incidents. We go through them one by one, and I test as many as I can against a real API server.

This is lesson 7 of the series (lessons run from 0 to 9). It builds on Configuration, Secrets and Storage in Kubernetes. The next lesson, Debugging Containers and Clusters, uses what we set up here: you will meet OOMKilled, Pending and failing probes as real symptoms. For the Deployment basics, see Kubernetes Core Ideas.

Words used in this lesson

• Probe is a check that the kubelet runs against a container to ask "are you alive?" or "are you ready?".
• Readiness probe decides if a pod receives traffic from Services.
• Liveness probe decides if a container must be restarted.
• Startup probe gives a slow-starting app extra time before the other probes begin.
• Request is the amount of CPU or memory the scheduler reserves for a container.
• Limit is the most CPU or memory a container may use.
• Throttling means slowing a process down because it used up its CPU allowance.
• OOM kill means the kernel stopped a process because it used more memory than allowed (OOM = out of memory).
• QoS class is a label (Guaranteed, Burstable, BestEffort) that decides which pods are removed first when a node is short of memory.
• Rolling update replaces pods a few at a time, so the app stays up.
• maxSurge / maxUnavailable are the two numbers that control how fast a rolling update goes.
• Rollback (undo) returns a Deployment to an older revision.
• HPA (HorizontalPodAutoscaler) changes the number of pod replicas based on a metric such as CPU.
• metrics-server is the add-on that gives the HPA its CPU and memory numbers.
• PodDisruptionBudget (PDB) says how many pods of an app must stay up during voluntary maintenance.
• Drain means moving all pods off a node before maintenance.
• Graceful shutdown means finishing in-flight requests when a pod is told to stop.

Running is not ready

Here is the problem in one picture. You deploy a Node.js API. The container starts in half a second, so Kubernetes says "Running". But your app needs 8 seconds to open its database connection and warm its cache. For those 8 seconds, any request sent to it fails.

If Kubernetes only knows "the process exists", it will send traffic too early. If your app later gets stuck in a deadlock (the process is alive but never answers), Kubernetes will not notice either.

Probes fix both. A probe is a small question the kubelet (the agent on each node) asks your container again and again. There are three:

The difference between readiness and liveness is the most asked probe question. Readiness is about traffic (take it out of rotation, it may come back). Liveness is about restart (a last resort). Remember lesson 5: a Service only sends traffic to ready pods, which is how a failing readiness probe removes a pod from the load balancing.

How a probe checks

A probe has one handler, which says how to ask:

• httpGet: the kubelet sends an HTTP GET. Any status code from 200 up to (but not including) 400 means success. Anything else is a failure.
• tcpSocket: the kubelet tries to open a TCP connection. If the port accepts it, that is success.
• exec: the kubelet runs a command inside the container. Exit code 0 is success.
• grpc: the kubelet calls the standard gRPC health-check service.

You can only give one handler per probe. The real API server enforces this. I put both httpGet and tcpSocket into a startupProbe, and it said:

The Deployment "notes-api" is invalid: spec.template.spec.containers[0].startupProbe.tcpSocket: Forbidden: may not specify more than 1 handler type

The timing fields and their defaults

I read these defaults from kubectl explain pod.spec.containers.livenessProbe on the real server:

So by default a liveness probe that fails three times in a row, ten seconds apart, restarts the container after about 30 seconds. Note the default timeoutSeconds: 1. If your health endpoint sometimes needs 1.2 seconds under load, it will fail the probe even though the app is fine. This is how a busy app gets killed by its own liveness probe.

The Deployment with all three probes

# deployment.yamlapiVersion: apps/v1kind: Deploymentmetadata:  name: notes-apispec:  replicas: 4  revisionHistoryLimit: 5  minReadySeconds: 5  progressDeadlineSeconds: 300  strategy:    type: RollingUpdate    rollingUpdate:      maxSurge: 1      maxUnavailable: 0  selector:    matchLabels:      app: notes-api  template:    metadata:      labels:        app: notes-api    spec:      terminationGracePeriodSeconds: 30      containers:        - name: api          image: notes-api:1.0          ports:            - name: http              containerPort: 3000          resources:            requests:              cpu: 250m              memory: 128Mi            limits:              memory: 256Mi          startupProbe:            httpGet:              path: /healthz              port: http            periodSeconds: 2            failureThreshold: 30          readinessProbe:            httpGet:              path: /ready              port: http            periodSeconds: 5            timeoutSeconds: 2            failureThreshold: 3          livenessProbe:            httpGet:              path: /healthz              port: http            periodSeconds: 10            timeoutSeconds: 2            failureThreshold: 3          lifecycle:            preStop:              exec:                command: ["sleep", "5"]

Let me explain each choice, because the choices are the lesson.

The startup probe checks /healthz every 2 seconds and allows 30 failures. That gives the app up to 2 x 30 = 60 seconds to start. During that time the liveness and readiness probes do not run, so a slow start is not mistaken for a stuck app. After the first success, the startup probe stops and the other two take over. This is better than a big initialDelaySeconds on the liveness probe, because a big delay wastes time on every fast start, and a short one kills slow starts.

The readiness probe uses a different path, /ready. This is on purpose. "Ready" can mean "my database connection works and my cache is warm". When the database blips, the app says "not ready" for a moment and drops out of the Service. It does not need a restart.

The liveness probe uses /healthz, which only answers "is my event loop alive?". It does not check the database.

Traps with probes

Also remember that probes cost something: they are real requests. A heavy /ready that runs a big database query every 5 seconds from 50 pods is real load.

Requests and limits: telling Kubernetes what you need

Each container can state two numbers for CPU and for memory.

• A request is what the scheduler reserves. When it picks a node, it adds up the requests of the pods already there, and only places your pod where there is room for its request.
• A limit is the ceiling that is enforced while the container runs.

CPU is measured in cores. 1 is one core, 500m is half a core (the m means thousandth, "millicores"). Memory is bytes with suffixes: 128Mi is 128 mebibytes, 1Gi is one gibibyte. (Mi and Gi are powers of 1024. 128M without the i means 128 million bytes, a slightly different number.)

The two resources behave very differently when the limit is reached. The Kubernetes docs say it clearly:

• A CPU limit is enforced by throttling. The container is slowed down, but it is not killed. CPU can be given back.
• A memory limit is enforced by the kernel with an OOM kill. Memory cannot be taken back from a running process, so the process is stopped. Kubernetes shows this as OOMKilled, exit code 137.

CPU throttling: I made it happen with Docker

Kubernetes sets CPU limits through a Linux feature called cgroups, and Docker uses the same feature with --cpus. I ran a busy loop for 5 seconds in a container limited to a quarter of a core, and read the throttling counters:

docker run --rm --cpus 0.25 mirror.gcr.io/library/alpine:3.20 sh -c '  timeout 5 sh -c "while :; do :; done"  grep -E "^(nr_periods|nr_throttled|throttled_time)" /sys/fs/cgroup/cpu/cpu.stat  cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us /sys/fs/cgroup/cpu/cpu.cfs_period_us'

Terminatednr_periods 54nr_throttled 50throttled_time 342217250525000100000

Read it like this. The kernel splits time into periods of 100,000 microseconds (the last line: 0.1 second). The container was allowed 25,000 microseconds of CPU per period (the line before: a quarter). Out of 54 periods, it was throttled in 50 of them, and in total it spent about 3.4 seconds (the number is in nanoseconds) waiting. The loop wanted a full core and got a quarter. (Your numbers will differ. This is cgroup v1 on my sandbox. On cgroup v2 the file is /sys/fs/cgroup/cpu.stat and uses throttled_usec.)

This is why a CPU limit can make a latency-sensitive API slow in bursts even when the node is mostly idle: in each 100 ms period the container may use up its quota in a few milliseconds and then must wait until the next period. Many teams set CPU requests but no CPU limits for this reason, and keep memory limits. I use that as my default: requests.cpu set, limits.memory set, no limits.cpu, unless a team has a reason (a noisy neighbour, a fair-share policy). It is an opinion, not a rule, so be ready to explain the trade-off in an interview.

Memory: the limit is a cliff

The memory limit is simple and brutal. If the container goes over, it is killed. Lesson 8 shows the Docker version with a real OOMKilled run (exit=137). In Kubernetes the pod's status then shows, in the words of the docs, Last State: Terminated, Reason: OOMKilled, Exit Code: 137, and the container restarts.

Two related facts. A memory request that is much lower than the real use causes trouble too: the scheduler packs many pods onto one node, and when the node itself runs out of memory, the kubelet evicts pods (see QoS below). And for Node.js, the V8 heap has its own default limit that depends on available memory, so set the container limit with that in mind. I would rather watch a few days of real usage than guess.

The API checks your numbers

I tried to request more memory than the limit:

requests:  memory: 512Milimits:  memory: 256Mi

The real API server answered:

The Deployment "notes-api" is invalid: spec.template.spec.containers[0].resources.requests: Invalid value: "512Mi": must be less than or equal to memory limit of 256Mi

QoS classes

From requests and limits Kubernetes works out a QoS class for each pod. When a node runs short of memory, it removes pods in a fixed order: BestEffort first, then Burstable, and Guaranteed last. I created three small pods and asked the real API server for their class:

# qos.yaml (three pods, shortened here)- name: q-guaranteed  resources:    requests: {cpu: 500m, memory: 256Mi}    limits:   {cpu: 500m, memory: 256Mi}- name: q-burstable  resources:    requests: {cpu: 100m, memory: 128Mi}    limits:   {memory: 256Mi}- name: q-besteffort  # no resources at all

NAME           QOSq-besteffort   BestEffortq-burstable    Burstableq-guaranteed   Guaranteed

The rule: Guaranteed means every container has CPU and memory requests equal to its limits. BestEffort means no requests or limits at all. Everything else is Burstable. Our notes-api Deployment (request 250m CPU, limit on memory only) is Burstable.

LimitRange and ResourceQuota: guard rails for a namespace

Two objects set rules for a whole namespace.

A LimitRange gives defaults to containers that say nothing. I created this one:

apiVersion: v1kind: LimitRangemetadata:  name: defaultsspec:  limits:    - type: Container      defaultRequest:        cpu: 100m        memory: 64Mi      default:        memory: 128Mi

Then I ran a pod with no resources in that namespace (kubectl run plain --image=notes-api:1.0) and asked what the pod got:

{"limits":{"memory":"128Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}Burstable

The defaults were added by the API server. The pod would have been BestEffort, and is Burstable now.

A ResourceQuota puts a cap on the total a namespace may use:

apiVersion: v1kind: ResourceQuotametadata:  name: team-quotaspec:  hard:    requests.cpu: "2"    requests.memory: 2Gi    limits.memory: 4Gi    pods: "20"

A quota has a rule that surprises people. When a quota covers a resource, every pod must state it. I made a Deployment with no resources in a namespace that had this quota. The ReplicaSet could not create its pod, and the real event said:

Warning  FailedCreate  replicaset/big-5f855ff4b7  Error creating: pods "big-5f855ff4b7-gpw7s" is forbidden: failed quota: team-quota: must specify limits.memory for: notes-api; requests.cpu for: notes-api; requests.memory for: notes-api

This is a famous "my Deployment shows 0/1 and no pods" mystery. The pods are not Pending. They are never created, so kubectl get pods shows nothing, and the answer is in kubectl describe replicaset or kubectl get events. A LimitRange with defaults avoids the problem.

What "Insufficient cpu" looks like

My fake node had 2 CPUs. I scaled the Deployment (each pod requests 250m) to 14 replicas. That asks for 3.5 CPUs. The scheduler placed what fit and left the rest Pending. The real event on a pending pod:

Warning  FailedScheduling  default-scheduler  0/1 nodes are available: 1 Insufficient cpu. no new claims to deallocate, preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

Read it as: "I looked at 1 node. It had no room for this pod's CPU request. Nothing could be removed to make room." Notice that this is based on requests, not on real usage. A node can look 5% busy in top and still be "full" for the scheduler, because the pods on it have reserved all its CPU. This is why honest requests matter: too high wastes money, too low causes overcrowded nodes and evictions.

Rolling updates: replacing pods without downtime

When you change the pod template of a Deployment (new image, new env var), the Deployment creates a new ReplicaSet and moves pods from the old one to the new one, a few at a time. Two numbers control the pace:

• maxSurge: how many pods above the desired count may exist during the update. Default 25%, rounded up.
• maxUnavailable: how many pods below the desired count may be missing. Default 25%, rounded down.

I confirmed the defaults on a plain Deployment from the real server: {"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"}, with revisionHistoryLimit 10 and progressDeadlineSeconds 600.

Percentages are confusing, so I wrote a tiny script to compute the limits. The rounding rules come from the Deployment docs.

// hpa-math.js (second half)function roll(replicas, surge, unavail) {  const up = (v) => (typeof v === "string" ? Math.ceil(replicas * parseInt(v) / 100) : v);  const down = (v) => (typeof v === "string" ? Math.floor(replicas * parseInt(v) / 100) : v);  return { maxPods: replicas + up(surge), minReady: replicas - down(unavail) };}console.log("10 replicas, defaults 25%/25%:", roll(10, "25%", "25%"));console.log("4 replicas, surge 1 / unavailable 0:", roll(4, 1, 0));console.log("3 replicas, defaults 25%/25%:", roll(3, "25%", "25%"));

10 replicas, defaults 25%/25%: { maxPods: 13, minReady: 8 }4 replicas, surge 1 / unavailable 0: { maxPods: 5, minReady: 4 }3 replicas, defaults 25%/25%: { maxPods: 4, minReady: 3 }

With 10 replicas and the defaults, there can be up to 13 pods during the rollout and at least 8 must stay ready. With only 3 replicas, 25% of 3 rounds down to 0 unavailable, so the update goes one pod at a time and never drops below 3 ready. Our Deployment says maxSurge: 1, maxUnavailable: 0. That is the safest setting for a small service: start one new pod, wait until it is ready, then stop one old pod. You always have all four ready pods. The price is that you need room for one extra pod, and the update is slower.

The two cannot both be zero. The real API server refused my attempt with maxSurge: 0 and maxUnavailable: 0:

The Deployment "notes-api" is invalid: spec.strategy.rollingUpdate.maxUnavailable: Invalid value: 0: may not be 0 when `maxSurge` is 0

(That makes sense: with no extra pod and no pod allowed to go missing, nothing could move.)

The probe is the gate

This is the key link between probes and rollouts. The Deployment counts a new pod as available only when it is ready, and it waits for minReadySeconds after that. It will not stop an old pod until the new one is available (with maxUnavailable: 0). So the readiness probe is what makes a rolling update safe. If the new version crashes or never gets ready, the rollout stops, and the old pods keep serving. A rolling update with a bad readiness probe, or none, can replace every good pod with a broken one.

Watching a rollout, and rolling back

The commands:

kubectl set image deployment/notes-api api=notes-api:1.1kubectl rollout status deployment/notes-apikubectl rollout history deployment/notes-apikubectl rollout undo deployment/notes-api

I ran these against the real control plane. I changed the image to 1.1 and then to a broken 1.2-broken, labelling each change with the annotation kubernetes.io/change-cause so the history is readable:

$ kubectl rollout history deployment/notes-apiREVISION  CHANGE-CAUSE1         <none>2         bump to 1.13         bump to 1.2 (broken)$ kubectl get rsNAME                   DESIRED   CURRENT   READY   AGEnotes-api-5ccddff559   1         1         0       8snotes-api-788b846476   3         3         0       44snotes-api-848f58fcd6   1         1         0       4s

You can see the idea. There are three ReplicaSets, one per revision. The old one (788b...) is being scaled down from 4 to 3, and new ones got one pod each. Then I asked for the status with a short timeout:

$ kubectl rollout status deployment/notes-api --timeout=8sWaiting for deployment "notes-api" rollout to finish: 1 out of 4 new replicas have been updated...error: timed out waiting for the condition

This output is real, but be careful with how I got it. No pod could ever become ready in my lab, because there is no kubelet. So this is exactly what a stuck rollout looks like: new pods that do not become ready, and a rollout that waits and does not make progress. The Deployment's conditions said:

Available=False MinimumReplicasUnavailableProgressing=True ReplicaSetUpdated

In a real cluster with a broken image, the same thing happens, and after progressDeadlineSeconds (600 seconds by default; I set 300) the condition changes to Progressing=False with reason ProgressDeadlineExceeded. Kubernetes does not roll back by itself. It only reports. Your pipeline or a tool such as Argo Rollouts has to decide to roll back.

The rollback:

$ kubectl rollout undo deployment/notes-apideployment.apps/notes-api rolled back$ kubectl rollout history deployment/notes-apiREVISION  CHANGE-CAUSE1         <none>3         bump to 1.2 (broken)4         bump to 1.1

Look at the history. The undo did not go "back to revision 2". It made a new revision, 4, that has the same pod template as old revision 2, and revision 2 disappeared from the list. (Revision 4 also shows the change-cause text bump to 1.1, copied from the old revision.) The Deployment's image was back to notes-api:1.1, which I checked with kubectl get deploy -o jsonpath. So history moves forward, and a rollback is just "re-apply an older template".

Other useful details from the real run and the docs:

• kubectl rollout restart deployment/NAME makes a new revision with only a restart annotation changed. It is how you restart all pods safely, using the same rolling pace. (I ran it; the history got a new, unlabelled revision.)
• kubectl rollout pause stops the Deployment from rolling out. While paused, I changed the image and no new ReplicaSet appeared. kubectl rollout resume lets you batch several edits into one rollout.
• revisionHistoryLimit (default 10) is how many old ReplicaSets are kept for rollback. Each costs a little memory in etcd. Setting it to 0 turns off rollback.
• kubectl rollout undo --to-revision=N goes to a specific revision.

The other strategy: Recreate

strategy.type: Recreate stops all old pods first and then starts the new ones. There is downtime. You use it only when two versions must never run at the same time, for example when the new version changes a database file format and the old version would corrupt it.

Graceful shutdown: the part people forget

A rolling update, a scale down and a node drain all end the same way for a pod: Kubernetes tells it to stop. What happens, in order, as described in the Pod lifecycle docs:

1. The pod is marked as terminating, and it is removed from the Service endpoints (so new traffic should stop coming).
2. If the container has a preStop hook, it runs.
3. The container gets SIGTERM.
4. After terminationGracePeriodSeconds (default 30), anything still running gets SIGKILL (exit code 137).

Two traps are inside this list. First, steps 1 and 3 happen at about the same time, and the load balancers or kube-proxy rules on other nodes can take a moment to catch up. So requests can still arrive for a short time after your app gets SIGTERM. That is why the Deployment above has preStop: sleep 5. The sleep keeps the app serving for five more seconds, while the endpoint removal spreads. It is a common, boring and effective fix.

Second, your app must handle SIGTERM: stop accepting new connections, finish the requests in flight, then exit. In Node.js that means calling server.close() in a process.on("SIGTERM", ...) handler. A container whose main process ignores SIGTERM waits the full grace period and is killed. Lesson 8 shows this with a real Docker run: a Node server with no handler took 10.2 seconds to stop and exited with code 137, and the same server with docker run --init stopped in 0.2 seconds. The Kubernetes version of the same problem is a rollout that takes 30 seconds per pod for no reason.

Autoscaling: more pods when load grows

The HorizontalPodAutoscaler (HPA) is a controller that watches a metric and changes replicas on a Deployment. This is the file:

# hpa.yamlapiVersion: autoscaling/v2kind: HorizontalPodAutoscalermetadata:  name: notes-apispec:  scaleTargetRef:    apiVersion: apps/v1    kind: Deployment    name: notes-api  minReplicas: 3  maxReplicas: 10  metrics:    - type: Resource      resource:        name: cpu        target:          type: Utilization          averageUtilization: 70  behavior:    scaleDown:      stabilizationWindowSeconds: 300      policies:        - type: Percent          value: 50          periodSeconds: 60    scaleUp:      stabilizationWindowSeconds: 0      policies:        - type: Pods          value: 4          periodSeconds: 60

The formula

The HPA runs a simple calculation, from the docs:

desiredReplicas = ceil( currentReplicas x currentMetricValue / desiredMetricValue )

"Utilization" for CPU means: the usage of the pods averaged, as a percentage of their requests. This is the reason pods must have a CPU request for a CPU-based HPA to work. With no request there is nothing to take a percentage of.

I wrote the formula as a function and ran it (with the 10% tolerance from the docs: if the ratio is within 10% of 1, nothing changes, which stops tiny wobbles from causing scaling):

current=4 metric=90% target=70% -> 6current=4 metric=75% target=70% -> 4current=4 metric=20% target=70% -> 2current=3 metric=300% target=70% -> 10

Line by line: 4 pods at 90% against a 70% target need ceil(4 x 90/70) = ceil(5.14) = 6. At 75% the ratio (1.07) is inside tolerance, so nothing happens. At 20% the raw answer is ceil(4 x 20/70) = 2. My script used a minimum of 1 for that line; in the real HPA above, minReplicas: 3 would hold it at 3. At 300% with 3 pods the raw answer is 13, but maxReplicas: 10 caps it at 10.

Scale up fast, scale down slowly

The behavior block tunes this. By default the HPA scales up quickly and scales down carefully: it looks at the highest recommendation of the last 300 seconds (5 minutes) before scaling down, so a short lull does not remove pods that you will need again a minute later. The policies above say: at most 4 pods added per minute, at most 50% of pods removed per minute. This prevents "flapping", the HPA repeatedly scaling up and down.

What the HPA does with no metrics: a real failure

The HPA needs the metrics-server add-on to read CPU and memory use. My lab has none. The HPA status I got from the real controller shows what a missing metrics-server looks like:

$ kubectl get hpaNAME        REFERENCE              TARGETS              MINPODS   MAXPODS   REPLICAS   AGEnotes-api   Deployment/notes-api   cpu: <unknown>/70%   3         10        0          6sConditions:  Type           Status  Reason                   Message  ----           ------  ------                   -------  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested ...

cpu: <unknown>/70% is the classic sign. The HPA can find the Deployment (AbleToScale True), but it cannot read metrics (ScalingActive False). The same lab also showed the matching client error: kubectl top pod printed error: Metrics API not available. If you ever see <unknown> in kubectl get hpa, check three things: is metrics-server installed and healthy, do the pods have CPU requests, and are the pods old enough to have a metric (it takes a minute or so).

HPA traps

PodDisruptionBudgets: keeping enough pods during maintenance

Pods can go away for two kinds of reasons. Involuntary disruptions are accidents: a node crashes, the hardware fails. Voluntary disruptions are actions people or tools take on purpose: draining a node for an upgrade, the cluster autoscaler removing a node, a rolling update of a node pool.

A PodDisruptionBudget (PDB) protects against voluntary ones. It says, for the pods that match a selector, how many must stay available, or how many may be down:

# pdb.yamlapiVersion: policy/v1kind: PodDisruptionBudgetmetadata:  name: notes-apispec:  minAvailable: 2  selector:    matchLabels:      app: notes-api

Now when someone runs kubectl drain node-1, the drain uses the Eviction API, which checks the PDB first. If removing a pod would leave fewer than 2 available, the eviction is refused and the drain waits and retries. A PDB does nothing against involuntary failures (a crashed node still takes its pods away). It only guards the polite actions.

I applied it. kubectl get pdb printed:

NAME        MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGEnotes-api   2               N/A               0                     6s

ALLOWED DISRUPTIONS 0 is real, and it is zero because none of my pods were ready. A PDB counts only pods that have the Ready condition. This shows the connection to probes: if your pods are not ready, the budget is empty, and drains are blocked.

Three more facts I tested or read:

• You can set minAvailable or maxUnavailable, not both. The real server said: minAvailable and maxUnavailable cannot be both set.
• A PDB with minAvailable equal to the number of replicas (or maxUnavailable: 0) means nothing can ever be evicted, and node upgrades will hang. This is a common mistake. With 2 replicas, minAvailable: 2 blocks every drain.
• I tried to evict one of my pods through the Eviction API while the budget showed 0 allowed. It succeeded ("status":"Success","code":201). That surprised me until I read the docs: pods in Pending, Succeeded or Failed phase are always allowed to be evicted, and my pods were Pending. The budget protects running pods. (There is also unhealthyPodEvictionPolicy. With the default IfHealthyBudget, a pod that is running but not ready can only be evicted if the app is not already below its budget, and the docs recommend AlwaysAllow so crash-looping pods do not block node drains.)

Putting it together: a checklist for a production Deployment

Before I call a Deployment "ready for production", I check these things:

1. Resources: CPU and memory requests based on measured use; a memory limit; a deliberate choice about the CPU limit.
2. Probes: a readiness probe that checks what the pod needs; a liveness probe that checks only the process; a startup probe if the start is slow.
3. Rollout: maxUnavailable: 0 and maxSurge: 1 (or a percentage), plus minReadySeconds if the app needs a settling time.
4. Shutdown: the app handles SIGTERM, and there is a preStop sleep or a grace period that is long enough for the longest request.
5. Replicas: at least 2 (so one pod can go), with a PDB (minAvailable lower than replicas), and ideally spread across nodes.
6. Scaling: an HPA with a sensible min and max, metrics-server installed, no replicas fight.

Every item has appeared in this lesson with a failure I could show or a doc I read.

Commands to know

Try it

1. A Deployment has 8 replicas and the default strategy. What is the most pods that can exist during an update, and the fewest that must be ready?
2. You set CPU requests of 500m and a pod uses 350m on average. What is its utilization for the HPA? If the target is 70%, what will the HPA do with 4 pods?
3. Your liveness probe calls /healthz, which runs SELECT 1 against the database. The database restarts for 20 seconds. What happens to your pods, and how do you fix the design?
4. A rollout hangs at "1 out of 4 new replicas have been updated". Name three places to look.
5. A node drain has been stuck for 30 minutes. The app has 2 replicas and a PDB with minAvailable: 2. Why, and what do you change?
6. On your own cluster: kubectl apply --dry-run=server a Deployment with a probe port name that does not exist. Does the server catch it? Then fix the YAML and run kubeconform -strict on it.

Answers

1. Surge 25% of 8 is 2 (rounded up), so at most 10 pods. Unavailable 25% of 8 is 2 (rounded down), so at least 6 must be ready.
2. Utilization is 350/500 = 70%. That is exactly the target, so the ratio is 1.0, inside tolerance, and the HPA changes nothing.
3. All pods fail liveness together after three failures (about 30 seconds with the defaults) and are restarted at once, maybe several times. Fix: liveness should only test the process (for example, return 200 if the event loop answers). Put the database check in readiness, so pods leave the Service but are not killed.
4. kubectl describe deployment (conditions), kubectl get rs and kubectl describe pod on a new pod (events: image pull, scheduling, probe failures), and kubectl logs of the new pod. Also check resource quota events on the ReplicaSet.
5. With 2 pods and minAvailable: 2, no pod may ever be evicted. Change to minAvailable: 1 (or maxUnavailable: 1), or run 3 or more replicas.
6. The server accepts it (configured (server dry run)), because the field is just a name. The error would only show up at runtime. kubeconform does not catch it either. You catch it by reading, or by running the Pod in a test cluster.

Interview questions

Key takeaways

• A running container is not a ready one. Use a readiness probe for traffic, a liveness probe only for "restart me", and a startup probe for slow starts.
• Never make liveness depend on the database or other pods.
• Requests are for scheduling. Limits are enforced: CPU by throttling, memory by OOM kill (137).
• QoS is worked out from requests and limits: Guaranteed, Burstable, BestEffort. Use LimitRange for defaults and ResourceQuota for caps. A quota makes every pod state its resources.
• "Insufficient cpu" is about requests, not real usage.
• A rolling update is controlled by maxSurge and maxUnavailable. Use maxUnavailable: 0 for zero downtime. The readiness probe is what makes it safe.
• Kubernetes reports a stuck rollout (ProgressDeadlineExceeded) but does not roll back by itself. rollout undo creates a new revision.
• Handle SIGTERM, and add a preStop sleep. Default grace period is 30 seconds.
• The HPA uses ceil(current x metric / target), needs CPU requests and metrics-server, and only adds pods, not nodes.
• A PDB guards voluntary disruptions only, counts only ready pods, and a too-strict one blocks node upgrades.

What's next

Next: Debugging Containers and Clusters, Lesson 8. Now that you know what healthy looks like, you will learn to read the symptoms of unhealthy: logs, exec, describe and events, and the classic failures CrashLoopBackOff, ImagePullBackOff, Pending and OOMKilled, many of them with real Docker runs. Practise now: add the three probes, requests, a memory limit and a PDB to one of your own Deployments and run kubeconform -strict on it.