Docker & Kubernetes
Kubernetes Core Ideas: Cluster, Pods and Deployments
Why orchestration, control plane and nodes, pods, ReplicaSets and Deployments, declarative YAML and the reconcile loop.
On this page
- Words used in this lesson
- Why we need orchestration
- The shape of a cluster
- Everything is an object in the API
- Pods
- ReplicaSets and Deployments
- The reconcile loop: how Kubernetes keeps its promise
- Rolling updates and rollbacks
- Declarative versus imperative
- Labels, selectors and namespaces
- Getting a cluster to practise on
- Try it
- Interview questions
- Key takeaways
- What's next
In one line: Kubernetes is a system where you write down what you want (for example "run 3 copies of this image"), save it in a central API, and a set of small programs called controllers work nonstop to make the real cluster match what you wrote. A Pod is the unit that runs your containers, a Deployment keeps the right number of Pods alive and rolls out new versions, and the loop that fixes differences is called reconciliation.
In lesson 3 Compose ran your API, Postgres and Redis on one machine. That is perfect on your laptop. In production a single machine is a single point of failure: if it dies, everything dies, and Compose will not move your containers anywhere else. Kubernetes (often written K8s) is built for that. It manages containers on many machines and keeps them running, even when machines and containers fail.
This lesson is the foundation for lessons 5 to 8. We cover the cluster, its parts, Pods, ReplicaSets, Deployments, and the idea that powers all of it. It is a lot of new words, so we take them one at a time. And I will not only describe them: I ran a real Kubernetes control plane on the sandbox, applied a Deployment to it, deleted Pods, stopped the controller, and recorded what happened. There is one important limit to that, explained in the note right below.
This is lesson 4 of 10. It builds on Why Containers (a Pod is built from the same namespaces) and Docker Compose. The next lesson, Services, Ingress and Networking, shows how Pods find each other. Then Configuration, Secrets and Storage, Scaling, Rollouts and Reliability, and Debugging follow.
What I ran / checked. I downloaded the real Kubernetes v1.37.1 server programs (kube-apiserver, kube-controller-manager, kube-scheduler) from the official release site (dl.k8s.io), plus etcd 3.6.4, and started them myself on one machine. kubectl is v1.37.1 too. So every kubectl result below about the API, Deployments, ReplicaSets, labels, events, validation errors, rollouts and rollbacks came from the real Kubernetes code. The one big fake: I could not run a real node. A normal node runs the kubelet, which starts containers. In this sandbox, a full kind cluster failed (its kubelet got runc create failed errors, nested containers do not work here). So I created two empty Node objects (worker-a, worker-b) and wrote a tiny Python script that pretends to be a kubelet. It marks the nodes Ready and, when a Pod has been assigned to a node, it edits the Pod's status to say Running. No container ever ran, no image was pulled, and kubectl logs and exec do not work (I tried: the real error is shown below). The scheduler, the Deployment controller and the ReplicaSet controller were all real, and they behaved exactly as they do in a cluster, but where I write "Running", remember the status was written by my script. For manifests, I validated with kubeconform -strict (v0.8.0) against the Kubernetes 1.34.0 schemas, yamllint, and kubectl apply --dry-run=server against my API server. I also made kubectl create ... --dry-run=client -o yaml write YAML. Sources I read (own words): the Kubernetes website docs (raw markdown from kubernetes/website) on Components, Pods, Deployments, ReplicaSets and Controllers, plus the kubectl explain text from the real API server. Could not reach: the rendered kubernetes.io site. I did not verify any GitHub star counts. Pod IPs shown (10.244.x.x) are numbers my script made up. Timings and generated names (like 78db6c85b9) change on every run.
Words used in this lesson
- Kubernetes is a system that runs and manages containers across many machines.
- Cluster is a set of machines running Kubernetes: a control plane plus worker nodes.
- Node is one machine (real or virtual) in the cluster that runs Pods.
- Control plane is the group of programs that decides what should happen in the cluster. It does not run your app.
- API server is the front door of the control plane. Everything (kubectl, controllers, nodes) talks through it.
- etcd is the database where the cluster keeps all its state.
- Scheduler is the program that picks a node for each new Pod.
- Controller is a program that watches objects and acts to make reality match the plan.
- kubelet is the agent on each node that starts and watches the containers of the Pods assigned to that node.
- Pod is the smallest thing Kubernetes runs: one or more containers that share a network and storage.
- ReplicaSet is an object that keeps a set number of identical Pods running.
- Deployment is an object that manages ReplicaSets, giving you scaling, rolling updates and rollbacks.
- Manifest is a YAML (or JSON) file that describes Kubernetes objects.
- Desired state is what you want. Current state is what exists now. A controller closes the gap.
- Label is a key and value tag on an object. A selector picks objects by label.
- Namespace is a named slice of the cluster, used to group and separate objects.
Why we need orchestration
Think about what you would have to do yourself if you had ten machines and thirty containers:
- Placement. Which machine has room for this container?
- Failure. A machine dies at night. Who starts its containers somewhere else?
- Scale. Traffic doubles. Who starts more copies, and later removes them?
- Updates. You ship version 2. How do you replace 10 copies without downtime?
- Finding each other. Containers move around. How does the API find the database?
- Config and secrets. How do you give each container its settings safely?
Compose answered a small version of 5 and 6 for one machine. An orchestrator answers all six. Kubernetes is the most widely used one. You do not tell it how. You tell it what you want, and it works out how.
The shape of a cluster
A Kubernetes cluster has two kinds of parts. The Kubernetes documentation lists them like this, and so will I.
YOU (kubectl, CI) | v +----------- CONTROL PLANE -----------+ | kube-apiserver <--> etcd | | kube-scheduler | | kube-controller-manager | +------------------+------------------+ | (nodes watch the API server) +-------------+-------------+ v v +-- NODE worker-a --+ +-- NODE worker-b --+ | kubelet | | kubelet | | kube-proxy | | kube-proxy | | container runtime | | container runtime | | [Pod] [Pod] | | [Pod] [Pod] | +-------------------+ +-------------------+Control plane (the brain, it makes decisions):
| Component | What it does |
|---|---|
kube-apiserver | Serves the Kubernetes HTTP API. Everything goes through it. It checks requests and stores objects. |
etcd | A consistent key-value database holding all cluster data. |
kube-scheduler | Looks for Pods with no node yet, and assigns each one to a suitable node. |
kube-controller-manager | Runs the built-in controllers (Deployment, ReplicaSet, Node, Job and others). |
cloud-controller-manager | Optional. Connects to a cloud provider for load balancers and so on. |
Node components (the hands, they do the work, on every worker machine):
| Component | What it does |
|---|---|
kubelet | Makes sure the containers of the Pods assigned to this node are running and healthy. |
| Container runtime | The program that actually runs containers (for example containerd). Lesson 0's namespaces and cgroups live here. |
kube-proxy | Maintains network rules so that Services work (lesson 5). Optional in some setups. |
The key design point: the control plane components do not talk to each other directly. They all talk to the API server. The scheduler does not tell the kubelet anything. It writes "Pod X goes to node B" into the API, and the kubelet on node B sees that and starts the Pod. This is called a watch: components subscribe to changes. This design is why Kubernetes survives failures well. Each part can restart and simply catch up by reading the current state from the API.
On a managed service (EKS, GKE, AKS) the cloud company runs the control plane for you. You only see your nodes. On your laptop, tools like kind, minikube or Docker Desktop run a small cluster.
What I ran, honestly
Here are the real programs I started on one machine for this lesson. (The 4 numbers in the left column are just process IDs.)
5044 dkstore (this is etcd 3.6.4, renamed so I could tell it apart from other lab processes)5112 dkapi (kube-apiserver v1.37.1)5603 dkcm (kube-controller-manager v1.37.1)5605 dksched (kube-scheduler v1.37.1)I confirmed the API server was healthy and the versions matched:
$ kubectl versionClient Version: v1.37.1Kustomize Version: v5.8.1Server Version: v1.37.1$ kubectl get --raw /readyzokAnd here is the fake part, made clear. My cluster has these nodes:
$ kubectl get nodesNAME STATUS ROLES AGE VERSIONworker-a Ready <none> 14mworker-b Ready <none> 14mNote the empty VERSION column. A real kubelet fills that in. My script does not. That is a small tell that these are not real machines. And because no kubelet exists:
$ kubectl logs notes-api-78db6c85b9-2bl9bError from server: no preferred addresses found; known addresses: []kubectl logs goes API server, then kubelet on the node, then container runtime. There is no kubelet to reach, so it fails. Everything that lives in the API (objects, scheduling, controllers) works. Anything that needs a real container does not.
Everything is an object in the API
In Kubernetes you work with objects: a Pod, a Deployment, a Service, a Node, a Namespace. Each is stored as data. My API server knows 71 resource types (kubectl api-resources counted them). The names I used in this lesson are all there: pods, deployments, replicasets, services, nodes, namespaces.
Where does the data live? In etcd. I asked etcd for the keys it stores (values are in a binary format, so I only listed names):
/registry/deployments/default/notes-api/registry/replicasets/default/notes-api-78db6c85b9/registry/replicasets/default/notes-api-85cfbcbcfd/registry/pods/default/notes-api-78db6c85b9-2bl9b/registry/pods/default/notes-api-78db6c85b9-cpjgb/registry/pods/default/notes-api-78db6c85b9-xx867One key per object. This is the full "database" of a cluster: if you back up etcd, you back up the cluster's whole state (this is why managed services do that for you).
The four parts of every object
Every Kubernetes manifest has the same four top-level fields:
apiVersion: apps/v1 # which API group and versionkind: Deployment # what type of objectmetadata: # name, namespace, labels name: notes-apispec: # what you WANT (desired state) replicas: 3and the cluster adds a fifth, status, which is what it currently has. You write spec. Kubernetes writes status. The whole system is about closing the gap between them.
You can ask Kubernetes to explain any field. kubectl explain reads the real API server's schema:
$ kubectl explain pod.spec.restartPolicyKIND: PodVERSION: v1FIELD: restartPolicy <string>ENUM: Always Never OnFailureDESCRIPTION: Restart policy for all containers within the pod. One of Always, OnFailure, Never. In some contexts, only a subset of those values may be permitted. Default to Always. ...This is a good habit: when you forget a field, ask kubectl explain instead of searching the web. (It needs a reachable API server, because it reads the server's schema.)
Pods
A Pod is the smallest thing Kubernetes can run. It is a wrapper around one or more containers. The Kubernetes docs say the shared context of a Pod is a set of Linux namespaces, cgroups and similar isolation: the same things that isolate a container. That is why lesson 0 came first.
Most Pods have one container. A Pod with several containers is for tightly coupled helpers: for example the app plus a log shipper. Containers in the same Pod:
- share one network namespace: the same IP address, and they talk to each other on
localhost, - can share volumes,
- are always scheduled on the same node, and start and stop together.
I can show this with plain Docker, because it uses the same trick. Docker lets a container join another container's network namespace with --network container:NAME:
docker run -d --name pause busybox:1.36 sleep 300docker run -d --name app --network container:pause busybox:1.36 sh -c 'httpd -f -p 8080 -h /etc'docker run --rm --network container:pause busybox:1.36 wget -qO- http://localhost:8080/hostnamefor c in pause app; do docker exec $c readlink /proc/self/ns/net; donefor c in pause app; do docker exec $c readlink /proc/self/ns/pid; donedocker exec app hostname -i ; docker exec pause hostname -i2fd35444d308net:[4026532269]net:[4026532269]pid:[4026532268]pid:[4026532324]172.17.0.2172.17.0.2The third container reached the web server in another container using localhost. Both containers show the same network namespace number (4026532269) and the same IP, but different PID namespaces. That is exactly the shape of a Pod: a shared network, separate processes. (Kubernetes runs a tiny hidden "pause" container per Pod to hold the namespaces, which is why I named mine pause. This is a model of it with Docker, not Kubernetes itself.)
Here is a Pod written as YAML, with an app container and a helper:
# pod.yamlapiVersion: v1kind: Podmetadata: name: notes-api-pod labels: app: notes-apispec: containers: - name: api image: notes-api:1.0 ports: - containerPort: 3000 env: - name: PORT value: "3000" - name: log-shipper image: busybox:1.36 command: ["sh", "-c", "sleep 3600"]Status: reviewed, not run with a real container. It passed kubeconform -strict (1.34.0 schemas) and yamllint, and kubectl apply --dry-run=server against my real API server said pod/notes-api-pod created (server dry run). When I really applied it, my fake kubelet marked it 2/2 Running on worker-b, but no container started. notes-api:1.0 is the image you would build in lessons 1 and 2, and it would have to be in a registry the cluster can reach. The containerPort is documentation only (like EXPOSE).
An environment variable's value must be a string, so "3000" is quoted. I tried it unquoted, and the real API server said: cannot unmarshal number into Go struct field EnvVar.spec.containers.env.value of type string.
Pods are mortal
Here is the thing that surprises everyone. A Pod created by itself is not replaced if it dies. I applied that Pod, then deleted it:
$ kubectl apply -f pod.yamlpod/notes-api-pod created$ kubectl delete pod notes-api-pod$ kubectl get pod notes-api-podError from server (NotFound): pods "notes-api-pod" not foundGone, and nothing brought it back. Also, in a real cluster, if the node dies, the Pods on it are lost. The docs call Pods "ephemeral", and add that you rarely create them directly. Instead you use a workload resource that creates and replaces Pods for you. The most common one is a Deployment.
Do not run "naked" Pods in production. A Pod with nobody managing it is not rescheduled when its node fails, and cannot be scaled or rolled out. Even a single-replica app should use a Deployment.
ReplicaSets and Deployments
A ReplicaSet has one job: keep N identical Pods running. You tell it how many, and a selector to find its Pods, and a template to build new ones. If a Pod disappears, it makes another. If there are too many, it deletes some.
A Deployment sits on top of ReplicaSets. The Kubernetes docs recommend using Deployments instead of managing ReplicaSets yourself, and say not to manage ReplicaSets that a Deployment owns. The Deployment adds the features you actually want: scaling, rolling updates, rollbacks and history.
The chain is:
Deployment -> ReplicaSet -> PodsEach arrow is an "owner" link. The Pod's metadata has an ownerReferences entry pointing to its ReplicaSet, and the ReplicaSet points to the Deployment. I read them from my cluster:
$ kubectl get pod notes-api-78db6c85b9-9cwdg -o jsonpath='{.metadata.ownerReferences}'[{"apiVersion":"apps/v1","blockOwnerDeletion":true,"controller":true,"kind":"ReplicaSet","name":"notes-api-78db6c85b9","uid":"930ff619-..."}]$ kubectl get rs -o jsonpath='{.items[0].metadata.ownerReferences[0].kind}/{.items[0].metadata.ownerReferences[0].name}'Deployment/notes-apiIf you delete the Deployment, Kubernetes follows the owner links and deletes the ReplicaSet and the Pods too (garbage collection).
Your first Deployment
This is the Deployment for our notes-api:
# deployment.yamlapiVersion: apps/v1kind: Deploymentmetadata: name: notes-apispec: replicas: 3 selector: matchLabels: app: notes-api template: metadata: labels: app: notes-api spec: containers: - name: api image: notes-api:1.0 ports: - containerPort: 3000Status: reviewed, not run with real containers. It passed kubeconform -strict and yamllint, and the real API server accepted it (deployment.apps/notes-api created). The Pods it produced were scheduled by the real scheduler, but "Running" was written by my fake kubelet.
How to read it:
replicas: 3is the desired number of Pods.selector.matchLabelsis how the Deployment finds its Pods: those with the labelapp: notes-api.templateis the blueprint for each Pod: its labels and itsspec(the containers). Note the nesting. Thetemplateis a Pod'smetadataandspecinside the Deployment'sspec.- The selector must match the template's labels. If it does not, the API rejects the Deployment. I tried it with a one-letter typo (
notes-apiiin the template):
$ kubectl apply -f dep-mismatch.yamlThe Deployment "notes-api" is invalid: spec.template.metadata.labels: Invalid value: {"app":"notes-apii"}: `selector` does not match template `labels`That is a real message from the real server, and a good one. Also notice that you cannot change a Deployment's selector afterwards. I tried that, too:
$ kubectl apply -f dep-selchange.yamlThe Deployment "notes-api" is invalid:* spec.template.metadata.labels: Invalid value: {"app":"web2"}: `selector` does not match template `labels`* spec.selector: Invalid value: {"matchLabels":{"app":"web"}}: field is immutableSo plan labels carefully: spec.selector is permanent.
Applying it
$ kubectl apply -f deployment.yamldeployment.apps/notes-api created$ kubectl get deploy,rs,pods -o wideNAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTORdeployment.apps/notes-api 0/3 3 0 4s api notes-api:1.0 app=notes-apiNAME DESIRED CURRENT READY AGE ... SELECTORreplicaset.apps/notes-api-78db6c85b9 3 3 0 4s app=notes-api,pod-template-hash=78db6c85b9NAME READY STATUS RESTARTS AGE IP NODE ...pod/notes-api-78db6c85b9-9cwdg 0/1 Pending 0 4s <none> <none>pod/notes-api-78db6c85b9-cpmxl 0/1 Pending 0 4s <none> <none>pod/notes-api-78db6c85b9-mrjbt 0/1 Pending 0 4s <none> <none>(I trimmed some columns.) This is the very first moment, before I had started my fake kubelet or created any nodes, so it shows the control plane working alone. In four seconds, with no node at all:
- The API server stored the Deployment.
- The Deployment controller created a ReplicaSet named
notes-api-78db6c85b9. The suffix is a hash of the Pod template. The docs say the Deployment controller puts thispod-template-hashlabel on each ReplicaSet and its Pods so that ReplicaSets of one Deployment never overlap. Do not change that label. - The ReplicaSet controller created 3 Pods.
- The scheduler looked at each Pod, found no nodes, and recorded an event:
Warning FailedScheduling pod/notes-api-78db6c85b9-mrjbt no nodes available to schedule podsNormal SuccessfulCreate replicaset/notes-api-78db6c85b9 Created pod: notes-api-78db6c85b9-9cwdgNormal ScalingReplicaSet deployment/notes-api Scaled up replica set notes-api-78db6c85b9 from 0 to 3The Pods sat in Pending. That is the right and honest state: they exist as objects, but nobody can run them. This is also the same state you see in a real cluster when it runs out of room (lesson 8). Then I started my script and it registered two nodes. A few seconds later:
$ kubectl get pods -o wideNAME READY STATUS RESTARTS AGE IP NODE notes-api-78db6c85b9-9cwdg 1/1 Running 0 23s 10.244.62.47 worker-anotes-api-78db6c85b9-cpmxl 1/1 Running 0 23s 10.244.194.125 worker-bnotes-api-78db6c85b9-mrjbt 1/1 Running 0 23s 10.244.99.19 worker-a$ kubectl get deployNAME READY UP-TO-DATE AVAILABLE AGEnotes-api 3/3 3 3 23sThe (real) scheduler placed two Pods on worker-a and one on worker-b. (The Running state and the IPs came from my script.) The columns of kubectl get deploy mean: READY is ready pods out of desired, UP-TO-DATE is pods that match the current template, AVAILABLE is pods that are ready for users.
The reconcile loop: how Kubernetes keeps its promise
Now the central idea. A controller is a loop that never ends:
- Look at the desired state (what the object's
specsays). - Look at the current state (what exists).
- If they differ, do something small to bring them closer.
- Repeat.
The Kubernetes docs use the thermostat example. You set 20 degrees (desired). The room is 17 (current). The thermostat turns on the heater. When it is 20, it turns off. It does not "run a heating program". It just compares and corrects, forever.
The docs also stress that a controller usually does not do the work itself. It sends messages to the API server (for example "create a Pod"), and other components act on that. That is exactly what we saw: the Deployment controller made a ReplicaSet object, the ReplicaSet controller made Pod objects, the scheduler set a node name, and the kubelet would start containers.
Let me prove it with four small experiments on the real controllers.
Experiment 1: delete a Pod
$ kubectl delete pod notes-api-78db6c85b9-9cwdgpod "notes-api-78db6c85b9-9cwdg" deleted from default namespace$ kubectl get pods (5 seconds later)NAME READY STATUS RESTARTS AGEnotes-api-78db6c85b9-cpmxl 1/1 Running 0 33snotes-api-78db6c85b9-g422j 1/1 Running 0 5snotes-api-78db6c85b9-mrjbt 1/1 Running 0 33sI removed a Pod. The ReplicaSet saw "desired 3, current 2" and created ...g422j within seconds. Compare this with the naked Pod that stayed deleted.
Experiment 2: change the desired number
$ kubectl scale deployment notes-api --replicas=5deployment.apps/notes-api scaled$ kubectl get pods --no-headers | wc -l5Then I scaled back to 3 and the ReplicaSet deleted two. The Deployment's events recorded it:
Normal ScalingReplicaSet deployment/notes-api Scaled up replica set notes-api-78db6c85b9 from 3 to 5Normal ScalingReplicaSet deployment/notes-api Scaled down replica set notes-api-78db6c85b9 from 5 to 3Experiment 3: change a label, and the Pod escapes
A ReplicaSet owns Pods by label. If I change a Pod's label so it no longer matches, the ReplicaSet no longer counts it, and makes a new one:
$ kubectl label pod notes-api-78db6c85b9-7z74h app=debug --overwrite$ kubectl get pods --show-labelsNAME READY STATUS LABELSnotes-api-78db6c85b9-7z74h 1/1 Running app=debug,pod-template-hash=78db6c85b9notes-api-78db6c85b9-blzxk 1/1 Running app=notes-api,pod-template-hash=78db6c85b9notes-api-78db6c85b9-cpjgb 1/1 Running app=notes-api,pod-template-hash=78db6c85b9notes-api-78db6c85b9-xx867 1/1 Running app=notes-api,pod-template-hash=78db6c85b9Now there are four Pods: three that belong to the Deployment, and the one I relabelled, which is no longer managed by anyone. A fourth was created to replace it. People use this trick to take a misbehaving Pod out of service for debugging while a replacement takes its place. It also shows why labels are the glue of Kubernetes: a Deployment does not hold a list of its Pods, it asks "who has these labels right now?".
Experiment 4: stop the controller
What if the controller itself is down? I stopped kube-controller-manager, then deleted a Pod:
$ kubectl delete pod notes-api-78db6c85b9-blzxkpod "notes-api-78db6c85b9-blzxk" deleted from default namespace$ kubectl get deploy notes-api ; kubectl get pods --no-headers | wc -lNAME READY UP-TO-DATE AVAILABLE AGEnotes-api 3/3 3 3 2m18s2Two Pods exist, but the Deployment's status still says 3/3, because no controller is awake to update it or to react. The Pod was not replaced. Then I started the controller manager again:
$ kubectl get pods --no-headers | wc -l (12 seconds later)3Three again. The controller started, compared desired (3) with current (2), and fixed it. This is the best way to understand the control plane: it is not a machine that does your work, it is a set of loops that fix differences. And if a loop is paused, the world is left as it was and is repaired when the loop returns. Real clusters run several copies of the control plane for this reason.
What about when a node fails? In a real cluster the node controller notices that a node stopped reporting, marks it not ready, and after a delay Pods on it are evicted and recreated elsewhere by their ReplicaSets. I did not simulate a node failure because my "nodes" are not real.
Rolling updates and rollbacks
You ship version 1.1 and need to replace the Pods. You change the image in the Deployment. I did it the quick way:
$ kubectl set image deployment/notes-api api=notes-api:1.1deployment.apps/notes-api image updated$ kubectl rollout status deployment/notes-apiWaiting for deployment "notes-api" rollout to finish: 1 out of 3 new replicas have been updated...Waiting for deployment "notes-api" rollout to finish: 2 out of 3 new replicas have been updated...Waiting for deployment "notes-api" rollout to finish: 1 old replicas are pending termination...deployment "notes-api" successfully rolled out$ kubectl get rsNAME DESIRED CURRENT READY AGEnotes-api-78db6c85b9 0 0 0 49snotes-api-85cfbcbcfd 3 3 3 4s(I shortened the rollout status lines.) A changed Pod template means a new ReplicaSet (new hash 85cfbcbcfd). The Deployment slowly scales the new one up and the old one down. Look at the events to see the steps:
Scaled up replica set notes-api-85cfbcbcfd from 0 to 1Scaled down replica set notes-api-78db6c85b9 from 3 to 2Scaled up replica set notes-api-85cfbcbcfd from 1 to 2Scaled down replica set notes-api-78db6c85b9 from 2 to 1Scaled up replica set notes-api-85cfbcbcfd from 2 to 3Scaled down replica set notes-api-78db6c85b9 from 1 to 0It moved one Pod at a time, so at every moment most of the Pods were still serving. The old ReplicaSet is kept at 0 replicas. That is your history, and it enables the rollback:
$ kubectl rollout history deployment/notes-apiREVISION CHANGE-CAUSE1 <none>2 <none>$ kubectl rollout undo deployment/notes-apideployment.apps/notes-api rolled back$ kubectl get deploy notes-api -o jsonpath='{.spec.template.spec.containers[0].image}'notes-api:1.0The Deployment went back to the old template and the old ReplicaSet. The "how many at once" settings are the defaults of the rolling update strategy. I read them back from the server:
$ kubectl get deploy notes-api -o jsonpath='{.spec.strategy}'{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"}maxSurge is how many extra Pods may exist above the desired count during the update. maxUnavailable is how many may be missing. Both default to 25 percent, rounded up for surge and rounded down for unavailable, as kubectl explain deployment.spec.strategy.rollingUpdate told me. The default history limit is 10 old ReplicaSets (revisionHistoryLimit).
In this lab "rolled out successfully" only means the Pods' status said Ready, which my script wrote for them. In a real cluster, the new Pods only count as ready when their containers run and pass readiness probes. Without a probe, Kubernetes marks a container ready as soon as it starts, even if your app is still booting. Lesson 7 covers probes and how a good rollout depends on them. Note also that the kubectl rollout undo warning I got was real: it said rolling back does not update the last-applied-configuration annotation, so keep your YAML file in Git as the source of truth.
Declarative versus imperative
There are two ways to use kubectl:
Imperative (you give commands): kubectl scale, kubectl set image, kubectl delete pod. Quick for experiments.
Declarative (you give a file): you edit deployment.yaml and run kubectl apply -f deployment.yaml. Kubernetes compares the file to the live object and changes only the difference. This is the way to work, because the file can live in Git, be reviewed and be repeated.
Let me change replicas: 3 to 4 in the file. First kubectl diff shows what would change, without changing it:
$ kubectl diff -f deployment.yaml- replicas: 3+ replicas: 4$ kubectl apply -f deployment.yamldeployment.apps/notes-api configured$ kubectl get deployNAME READY UP-TO-DATE AVAILABLE AGEnotes-api 4/4 4 4 14m(I removed lines that only show bookkeeping fields, like generation.) Applying an unchanged file prints unchanged, which is the idempotent behaviour you want in CI. Two other habits:
kubectl apply -f deployment.yaml --dry-run=server # ask the real API server if it would accept itkubectl create deployment web --image=nginx:1.27 --replicas=2 --dry-run=client -o yamlThe second one writes YAML for you without creating anything. Here is the start of what it printed:
apiVersion: apps/v1kind: Deploymentmetadata: labels: app: web name: webspec: replicas: 2 selector: matchLabels: app: web strategy: {}That is a good way to get a skeleton, then edit it. (It puts the same label on the selector and the template for you, so the mismatch error cannot happen.)
Mistakes the API server catches
I tried some deliberate typos so you can see real errors:
# "replica:" instead of "replicas:"$ kubectl apply -f dep-typo2.yamlError from server (BadRequest): error when creating "dep-typo2.yaml": Deployment in version "v1" cannot be handled as a Deployment: strict decoding error: unknown field "spec.replica"# an API version that does not exist in this cluster$ kubectl apply -f dep-oldapi.yamlerror: resource mapping not found for name: "notes-api" namespace: "" from "dep-oldapi.yaml": no matches for kind "Deployment" in version "apps/v1beta1"ensure CRDs are installed firstThe first one is strict: modern kubectl and the server reject unknown fields. And here is a trap: with --validate=false the same typo went through, and the Deployment got the default of 1 replica, silently:
$ kubectl apply --validate=false -f dep-typo2.yamldeployment.apps/typo-test created$ kubectl get deploy typo-test -o jsonpath='{.spec.replicas}'1So never turn validation off to "make an error go away". A tab character in YAML is another classic: kubectl reported found character '\t' that cannot start any token. YAML only allows spaces for indentation.
You can also check files before you have any cluster. kubeconform -strict compared my two manifests with the 1.34 schemas and reported Valid: 2, Invalid: 0. Given the typo file, it reported the field as invalid too. Note: it cannot catch everything. It will accept a Deployment whose labels do not match anything you meant, so it is a first filter, not a test.
Labels, selectors and namespaces
Labels are small key and value tags (app: notes-api, tier: backend, env: prod). Selectors pick objects by label. You have already seen two uses: the Deployment finds its Pods with a selector, and kubectl get pods -l app=notes-api filters by label. On my cluster:
$ kubectl get pods -l app=notes-api(3 pods)$ kubectl get pods -l app=nothingNo resources found in default namespace.In lesson 5 a Service will use a selector to find the Pods it sends traffic to. The idea is always the same: "all objects that look like this".
A namespace is a named group of objects inside one cluster. A new cluster starts with default, kube-system, kube-public and kube-node-lease. Names only have to be unique inside a namespace, so you can run the same app twice. I applied the same file into a new namespace:
$ kubectl create namespace stagingnamespace/staging created$ kubectl apply -n staging -f deployment.yamldeployment.apps/notes-api created$ kubectl get deploy -Adefault notes-api 3/3staging notes-api 3/3Two independent notes-api Deployments, with the same name, side by side. Teams use namespaces to separate environments or teams, and to attach quotas and permissions. A namespace is not a security wall by itself: with the default network settings, Pods in different namespaces can still talk to each other (lesson 5 covers NetworkPolicy).
Getting a cluster to practise on
You do not need a cloud account.
- Docker Desktop has a one-click Kubernetes option.
- kind ("Kubernetes in Docker") runs each node as a Docker container. Good for CI and testing.
- minikube runs a small cluster on a VM or container.
- A managed cluster from a cloud (EKS, GKE, AKS) costs money but is real.
Check with kubectl config current-context which cluster kubectl is pointing at before you apply anything. Applying to the wrong context is a very common and expensive mistake.
In this sandbox kind could not run (nested containers failed at the runtime level), which is why I built the control plane from the release binaries. On a normal laptop with Docker, kind create cluster works, and you would run the same commands from this lesson against real Pods.
Try it
You can do these on kind, minikube or Docker Desktop. The answers are what a real cluster does.
- Apply the Deployment above with the image
nginx:1.27andcontainerPort: 80. Runkubectl get pods -win one terminal. In another, delete one Pod. What do you see? - Which two things must match in a Deployment, or the API rejects it?
- What is the difference between
kubectl apply -f x.yamlandkubectl run? - You change
spec.template.spec.containers[0].image. Which new object appears? Which command shows the old one is kept? - A teammate says "I changed the replica count in the file but nothing happened." What did they forget?
- Why is
kubectl delete podon a Deployment's Pod usually safe, but on a naked Pod not? - What is in
status, and who writes it?
Answers
- The deleted Pod goes to
Terminating, and almost immediately a new Pod with a different random suffix appears and becomesRunning. The ReplicaSet restores the count. spec.selector.matchLabelsmust match the labels inspec.template.metadata.labels. (Also, the selector cannot be changed afterwards.)apply -fis declarative: it makes the cluster match the file and can be repeated.kubectl runcreates a single Pod right now, with no controller, so it is not rescheduled if it dies.- A new ReplicaSet appears (a new hash).
kubectl get rsshows the old one with 0 replicas, andkubectl rollout historylists revisions. - They did not run
kubectl apply -fafter saving. The file on disk is not the cluster. - A Deployment's ReplicaSet notices the missing Pod and creates another. A naked Pod has no controller, so it is simply gone.
statusis the observed current state (ready replicas, conditions, Pod IPs). The controllers and kubelets write it. You writespec.
Interview questions
What problems does Kubernetes solve that Docker or Compose do not? (Must know)
Compose runs containers on one machine. Kubernetes runs them across many machines and keeps them running when machines fail. It handles placement (scheduling), self-healing (replace dead Pods), scaling, rolling updates and rollbacks, service discovery, and configuration. The key is the declarative model: I state the desired result and controllers keep making the cluster match it.
Name the parts of the control plane and what each does. (Must know)
The API server is the front door and validates and stores objects. etcd is the database of all cluster state. The scheduler assigns unscheduled Pods to nodes. The controller manager runs the controllers, such as the Deployment, ReplicaSet and Node controllers. On each node, the kubelet runs the Pods assigned to it, the container runtime runs the containers, and kube-proxy sets up Service networking. The components talk through the API server, not directly to each other.
What is a Pod, and why isn't the container the unit? (Must know)
A Pod is one or more containers that share a network namespace (one IP, localhost between them), can share volumes, and are always placed on the same node. It is the unit of scheduling and scaling. The extra layer lets tightly-coupled helpers, like a log shipper, live next to the app. I demonstrated the shape with Docker's --network container: option: two containers with the same network namespace but different PID namespaces.
Explain the relationship between a Deployment, a ReplicaSet and a Pod. (Must know)
A Deployment owns ReplicaSets and a ReplicaSet owns Pods, linked by ownerReferences. The ReplicaSet keeps N Pods matching a template. The Deployment manages ReplicaSets so it can do rolling updates and rollbacks: a new Pod template creates a new ReplicaSet, which is scaled up while the old one is scaled down, and the old one is kept for rollback. In my test the hash in the names (78db6c85b9, 85cfbcbcfd) identified each template.
What is the reconcile loop? (Must know)
A controller repeatedly compares the desired state in an object's spec with the observed state, and takes small steps to remove the difference. It is level-based, so it works from the current state and not from a list of events it might have missed. I showed it three ways: a deleted Pod was recreated, a relabelled Pod caused a replacement, and when I stopped the controller manager a deleted Pod stayed missing until the controller came back and repaired it.
What happens, step by step, when you run kubectl apply -f deployment.yaml? (Senior signal)
kubectl sends the object to the API server, which authenticates, validates and stores it in etcd. The Deployment controller sees the new Deployment and creates a ReplicaSet. The ReplicaSet controller sees it and creates the Pods. The scheduler sees Pods without a node and picks a node for each, writing it into the Pod. The kubelet on that node sees a Pod assigned to it, tells the container runtime to pull the image and start the containers, and reports status back to the API server. Each step is a separate component watching the API, which I saw in the events: ScalingReplicaSet, SuccessfulCreate, then scheduling.
What does a Pending Pod mean and what are common causes? (Senior signal)
Pending means the Pod exists but is not running yet. Common causes: the scheduler cannot find a node with enough free CPU or memory, node selectors or taints do not match, a volume cannot be bound, or the image is still being pulled. kubectl describe pod shows the events. In my lab the Pods stayed Pending with FailedScheduling: no nodes available until I registered nodes, and that is the same state you get when a real cluster has no room. Lesson 8 covers this routine.
What is the difference between declarative and imperative management? Which do you prefer? (Must know)
Imperative means telling Kubernetes to do actions (kubectl scale, kubectl delete). Declarative means keeping manifests that describe the end state and applying them (kubectl apply -f). I prefer declarative for anything real, because the files live in Git, can be reviewed, and re-applying them is safe. I use imperative commands for quick experiments and debugging. And I use kubectl diff and --dry-run=server before applying.
Why must a Deployment's selector match its template labels, and what happens if you change it later? (Trap)
The selector is how the Deployment and its ReplicaSets identify the Pods they own, so the template must produce Pods that match, or the Deployment would create Pods it cannot find. The API rejects a mismatch, and I got the real error. The selector is also immutable, so changing it fails with field is immutable. To change it you have to delete and recreate the Deployment. Also, overlapping selectors between two controllers can make them fight over the same Pods.
How does a rolling update work, and how do maxSurge and maxUnavailable change it? (Senior signal)
A new template creates a new ReplicaSet. The Deployment scales it up and the old one down in steps. maxSurge is how many Pods above the desired count may exist, and maxUnavailable is how many below it may be missing. Both default to 25 percent. In my test with 3 replicas it moved one Pod at a time. The quality of the rollout depends on readiness probes. Without them a Pod counts as ready when it starts, which can send traffic to an app that is not ready.
How do you roll back a bad release? (Must know)
kubectl rollout undo deployment/NAME returns to the previous ReplicaSet, and --to-revision=N picks a specific one. kubectl rollout history lists revisions. This works because the Deployment keeps old ReplicaSets (10 by default). A caution I saw first hand: kubectl warned that undo does not update the last-applied-configuration annotation, so I keep the YAML in Git as the source of truth and prefer reverting the file and re-applying.
What is a namespace for, and is it a security boundary? (Trap)
A namespace groups objects and scopes names, so teams or environments can share a cluster. It is also where you attach quotas, RBAC rules and network policies. By itself it is not a security boundary: with the default network, Pods in different namespaces can still talk, and cluster-wide objects (like nodes) are not in any namespace. I run the same Deployment in two namespaces to show that names only need to be unique within one.
Key takeaways
- Kubernetes keeps your desired state (what you wrote in
spec) equal to the current state (status) using controllers that loop forever. - The control plane is the API server, etcd, scheduler and controller manager. Nodes run the kubelet, a container runtime and kube-proxy. Parts talk through the API server only.
- A Pod is containers that share a network, volumes and a node. A naked Pod is not replaced when it dies.
- A Deployment owns a ReplicaSet, which owns Pods. A new template makes a new ReplicaSet, and the old one is kept for rollbacks.
- Deleting a Pod from a Deployment is safe because the controller replaces it. When I stopped the controller, nothing was replaced until it returned.
- The Deployment
selectormust match the template labels, and it is immutable. - Work declaratively:
kubectl apply -f,kubectl diff,--dry-run=server. Do not disable validation. - Check your context before you apply, and keep YAML in Git.
What's next
Next: Services, Ingress and Networking in Kubernetes, Lesson 5. You now have three Pods that can be replaced at any time, and each has a different IP. The next lesson solves the obvious problem: how does anything find them? You will meet the Service, stable names with DNS, and Ingress for traffic from outside. Practise now: install kind or Docker Desktop's Kubernetes, apply the Deployment above with nginx, delete a Pod, and watch it come back.
Published by Jainik · Last updated · How these lessons are made
Keep learning
Related lessons
- Containers, Kubernetes, Terraform and AWS for Full-Stack Interviews · Full-Stack Questions
Part of the study guide
Related series