Docker & Kubernetes

Configuration, Secrets and Storage in Kubernetes

ConfigMaps, Secrets and their limits, persistent volumes and claims, StatefulSets at a glance.

Lesson 0614 min read#kubernetes#configmap#secrets#storage

On this page

  1. Words used in this lesson
  2. The idea: one image, many environments
  3. ConfigMaps: settings as an object
  4. Three ways to use a ConfigMap in a pod
  5. Secrets: the same shape, a different intent
  6. Volumes: giving a pod a folder
  7. PersistentVolumes and claims: the two-sided deal
  8. StatefulSets: pods with names and their own disks
  9. Commands to know
  10. Try it
  11. Interview questions
  12. Key takeaways
  13. What's next

In one line: a good image has no settings inside it. Kubernetes gives you a ConfigMap for normal settings, a Secret for sensitive ones (but base64 is not encryption), and volumes for data that must survive a restart. Together they let you run the same image in development and in production, and not lose your data when a pod dies.

In the last lesson your pods learned to find each other. Now they need three more things. They need settings (which log level? which database host?). They need passwords (the database password, an API key). And some of them need disk space that does not disappear when the pod is replaced.

Kubernetes has a separate object for each of these. This lesson explains them one by one, with small examples. We also open the box and look at where a Secret really lives, because that is the part most people get wrong.

This is lesson 6 of the series (lessons run from 0 to 9). It builds on Services, Ingress and Networking in Kubernetes. The next lesson, Scaling, Rollouts and Reliability, makes these pods safe to update and scale. If you want the Docker-only version of this topic first, Volumes, Networks and Running Containers Safely covers volumes, bind mounts and environment variables on one machine.

What I ran / checked. Docker Engine 29.8.2 has a real daemon in my sandbox. kubectl v1.34.0, kubeconform v0.7.0 (checks YAML against the official Kubernetes 1.34.0 JSON schemas), yamllint 1.38.0 and Node v22.22.0 are real too. There is no cluster node in the sandbox (the kind approach failed, as the previous lesson explains), so I did something different this time. I downloaded the real Kubernetes v1.34.0 kube-apiserver, kube-controller-manager and kube-scheduler binaries, plus etcd 3.5.21 (the database behind the API server), and started them by hand on localhost. That gives me a real control plane with no kubelet and no worker node. So the API server really accepted, validated, stored and rejected my objects, the controllers really created ReplicaSets and PVC events, and I could read the stored bytes straight from etcd. But no container ever ran on Kubernetes. I registered one fake Node object by hand so the scheduler had something to talk to. Because of that, anything that needs a kubelet (a pod becoming Running, a mounted ConfigMap file appearing in a container, a volume really attaching) is labelled "reviewed, not run on a cluster here", and I did not paste invented output for it. Sources I read (own words below): the Kubernetes website docs for ConfigMaps, Secrets, Persistent Volumes, StorageClasses and StatefulSets (raw markdown from kubernetes/website on GitHub), and kubectl explain output from the real API server. Could not reach: the rendered kubernetes.io site. I verified no GitHub star counts.

Words used in this lesson

  • ConfigMap is an object that stores non-secret settings as key and value pairs.
  • Secret is an object that stores sensitive values (passwords, tokens, keys). It looks like a ConfigMap but is handled with more care.
  • base64 is a way to write bytes as plain letters. It is not encryption. Anyone can reverse it.
  • Environment variable is a named value that a process reads when it starts, like LOG_LEVEL=info.
  • Volume is a folder that is mounted into a container. In Kubernetes it belongs to the pod, not to one container.
  • emptyDir is a scratch volume that is created empty with the pod and deleted with the pod.
  • PersistentVolume (PV) is a piece of real storage in the cluster, such as a cloud disk.
  • PersistentVolumeClaim (PVC) is a pod's request for storage: "I need 5 GiB, read-write". Kubernetes finds or creates a PV to match.
  • StorageClass is a recipe for making storage on demand, such as "fast SSD, delete when released".
  • Provisioner is the program that creates the real disk for a StorageClass.
  • Access mode says who may mount a volume, for example ReadWriteOnce (one node at a time).
  • StatefulSet is like a Deployment, but each pod gets a stable name and its own disk.
  • etcd is the database where Kubernetes stores every object, including Secrets.
  • Encryption at rest means data is encrypted on disk, so that a stolen disk or backup is not readable.

The idea: one image, many environments

In lesson 1 we built an image. The rule for a good image is: build once, run anywhere. The same image should run on your laptop, in testing and in production. If you bake the database address into the image, you need a different image for each place. That is slow, and you are no longer testing what you ship.

So the image contains the code. The settings come from outside, when the container starts. That is what ConfigMaps and Secrets are for.

What it isExampleWhere it lives
Code and fixed filesserver.js, node_modulesInside the image
Normal settingsLOG_LEVEL=info, PAGE_SIZE=20ConfigMap
Sensitive settingsdatabase password, API keySecret
Data that changes and must be keptuploaded files, database filesVolume (PVC)

ConfigMaps: settings as an object

A ConfigMap is an object with a data section. The data is just names and values (strings). The name of the ConfigMap and the keys inside it are for you to choose.

There are three easy ways to create one. Let me use kubectl create ... --dry-run=client -o yaml. This does not talk to a cluster. It only prints the YAML that kubectl would send. First I made a small file app.json with one line, then I ran:

Shell
kubectl version --clientkubectl create configmap notes-config \  --from-literal=LOG_LEVEL=info \  --from-literal=PAGE_SIZE=20 \  --from-file=app.json=files/app.json \  --dry-run=client -o yaml
TEXT
Client Version: v1.34.0Kustomize Version: v5.7.1apiVersion: v1data:  LOG_LEVEL: info  PAGE_SIZE: "20"  app.json: |    { "pageSize": 20, "cacheSeconds": 60 }kind: ConfigMapmetadata:  name: notes-config

Look at what happened. --from-literal made one short key per flag. --from-file made a key called app.json whose value is the whole content of the file. A ConfigMap does not care if a value is one word or a whole JSON file. Also notice that PAGE_SIZE got quotes: "20". That is on purpose, and it leads to our first trap.

Trap: every value in a ConfigMap is a string

Here is a ConfigMap that looks fine to a human:

YAML
apiVersion: v1kind: ConfigMapmetadata:  name: notes-configdata:  PAGE_SIZE: 20  DEBUG: true

YAML reads 20 as a number and true as a boolean. A ConfigMap wants strings. I checked this file with kubeconform (strict mode, Kubernetes 1.34.0 schemas):

TEXT
bad-number.yaml - ConfigMap notes-config is invalid: problem validating schema. Check JSON formatting: jsonschema validation failed with 'https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.34.0-standalone-strict/configmap-v1.json#' - at '/data/DEBUG': got boolean, want null or string - at '/data/PAGE_SIZE': got number, want null or string

The real API server says the same thing when you try to apply it (the object already existed in my test, so the message is about a patch):

TEXT
The request is invalid: patch: Invalid value: "map[data:map[DEBUG:true ...]]": cannot convert int64 to string

The fix is to put quotes around every value: PAGE_SIZE: "20", DEBUG: "true". Your app then converts the string to a number or boolean itself. In Node.js that means Number(process.env.PAGE_SIZE). Remember that environment variables are always strings, even in Docker.

Quote your numbers and booleans. Unquoted 20 and true in a ConfigMap (and in an env value) are rejected. Also yes, no and on can be turned into booleans by some YAML parsers. When in doubt, quote it.

The ConfigMap file I use for the rest of this lesson:

YAML
# configmap.yamlapiVersion: v1kind: ConfigMapmetadata:  name: notes-configdata:  LOG_LEVEL: "info"  PAGE_SIZE: "20"  app.json: |    { "pageSize": 20, "cacheSeconds": 60 }

Status: applied to a real v1.34.0 API server (no kubelet), so it was accepted and stored. It also passed kubeconform -strict -kubernetes-version 1.34.0 and yamllint. kubectl get configmap then showed notes-config 3 in the DATA column, so the three keys were stored.

Two limits to know. A ConfigMap can hold at most 1 MiB of data. And a ConfigMap is not secret. The Kubernetes docs say this in a warning box: a ConfigMap gives no secrecy or encryption. Passwords go in a Secret (next section).

Three ways to use a ConfigMap in a pod

The ConfigMap by itself does nothing. A pod has to refer to it. The pod and the ConfigMap must be in the same namespace. There are three common ways to use it.

1. One key as one environment variable (valueFrom with configMapKeyRef):

YAML
env:  - name: LOG_LEVEL    valueFrom:      configMapKeyRef:        name: notes-config        key: LOG_LEVEL

2. All keys as environment variables (envFrom). Every key in the ConfigMap becomes a variable. This is short, but you can pull in names you did not expect, so I use it only for small ConfigMaps that I own.

YAML
envFrom:  - configMapRef:      name: notes-config

3. Keys as files (a volume). Each key becomes a file, and the value becomes the file content. This fits config files like app.json or nginx.conf.

YAML
volumeMounts:  - name: config    mountPath: /etc/notes    readOnly: truevolumes:  - name: config    configMap:      name: notes-config      items:        - key: app.json          path: app.json

Without items, every key becomes a file (here: LOG_LEVEL, PAGE_SIZE and app.json). With items, only the keys you list become files, and path is the file name. So this mount gives your app one file: /etc/notes/app.json.

Here is a full Deployment that uses all of this, plus a Secret. It is the notes-api Deployment from the earlier lessons, with config added.

YAML
# deployment.yamlapiVersion: apps/v1kind: Deploymentmetadata:  name: notes-apispec:  replicas: 2  selector:    matchLabels:      app: notes-api  template:    metadata:      labels:        app: notes-api    spec:      containers:        - name: api          image: notes-api:1.0          ports:            - name: http              containerPort: 3000          env:            - name: LOG_LEVEL              valueFrom:                configMapKeyRef:                  name: notes-config                  key: LOG_LEVEL            - name: DB_PASSWORD              valueFrom:                secretKeyRef:                  name: notes-db                  key: DB_PASSWORD          envFrom:            - secretRef:                name: notes-db                optional: false          volumeMounts:            - name: config              mountPath: /etc/notes              readOnly: true      volumes:        - name: config          configMap:            name: notes-config            items:              - key: app.json                path: app.json

Status: the API server accepted this Deployment, and the Deployment controller created a ReplicaSet and two pods. The pods stayed Pending because there was no kubelet. It passed kubeconform -strict and yamllint. What the app would then see inside a container (environment variables and /etc/notes/app.json) is reviewed, not run on a cluster here.

What happens when the ConfigMap changes?

This is a very common interview trap, so slow down here. The Kubernetes docs say:

  • A ConfigMap used as files (a volume) is updated eventually. The kubelet checks it on its normal sync loop, so the delay can be around a minute. Your app has to re-read the file to notice the change.
  • A ConfigMap used as environment variables is never updated in a running container. The environment is fixed when the container starts. You need a new pod.
  • A ConfigMap mounted with subPath also never gets updates.
How it is usedUpdates in a running pod?
Environment variable (env, envFrom)No. Restart the pod.
Volume fileYes, after a delay. The app must re-read the file.
Volume with subPathNo.

(This table is from the Kubernetes docs. I could not watch it happen, because it needs a kubelet. Reviewed, not run on a cluster here.)

So how do you roll out a config change? The simplest way is to make Kubernetes replace the pods:

Shell
kubectl rollout restart deployment/notes-api

The pods are replaced one by one (a rolling update, which we study in the next lesson), and the new pods read the new values. Another popular way is to put a hash of the config into a pod annotation. When the config changes, the hash changes, the pod template changes, and Kubernetes starts a rollout by itself. Tools like Helm and Kustomize do this for you.

Immutable ConfigMaps

You can mark a ConfigMap (or a Secret) as immutable: true. After that, nobody can change its data. You must make a new one with a new name. The benefits: nobody can break running pods by a bad edit, and the cluster has less work to do, because the kubelet does not need to watch it for changes.

YAML
apiVersion: v1kind: ConfigMapmetadata:  name: notes-config-v2immutable: truedata:  LOG_LEVEL: "debug"

I created this and then tried to change debug to info. The real API server said:

TEXT
The ConfigMap "notes-config-v2" is invalid: data: Forbidden: field is immutable when `immutable` is set

The usual pattern is to put a version or hash in the name (notes-config-v2) and point the Deployment at the new name. That change to the pod template triggers a normal rolling update, and rolling back is just pointing at the old name again.

Secrets: the same shape, a different intent

A Secret looks almost the same as a ConfigMap. It has a data section. The difference is the intent, and a few protections. Kubernetes can handle Secrets more carefully (for example, a kubelet keeps secret volumes in memory on the node, not on disk, and the API server can encrypt them at rest if you turn that on). You can also give different people different permissions on Secrets than on ConfigMaps.

Let me create one and look at it:

Shell
kubectl create secret generic notes-db \  --from-literal=DB_USER=notes \  --from-literal=DB_PASSWORD='s3cr3t-pw' \  --dry-run=client -o yaml
TEXT
apiVersion: v1data:  DB_PASSWORD: czNjcjN0LXB3  DB_USER: bm90ZXM=kind: Secretmetadata:  name: notes-db

The values look scrambled. They are base64. Let me reverse one:

Shell
echo 'czNjcjN0LXB3' | base64 -d
TEXT
s3cr3t-pw

So anyone who can read the Secret object can read the password. The Kubernetes docs even say that base64 only obscures the value, it does not make it secret. The reason Secrets use base64 is simple: a value may be binary (a certificate, a key file), and base64 lets it fit inside YAML and JSON. It is a format, not a lock.

base64 is not encryption. If someone says "we are safe, the Secret is base64", they are wrong. Safety comes from who is allowed to read the object (RBAC, which we cover below), and whether the data is encrypted where it is stored.

data and stringData

In a Secret you may write values in two sections. data wants base64 text. stringData takes plain text, and the API server converts it for you. stringData is easier to write by hand:

YAML
# secret.yamlapiVersion: v1kind: Secretmetadata:  name: notes-dbtype: OpaquestringData:  DB_USER: notes  DB_PASSWORD: s3cr3t-pw

I applied this. The stored object had only data (with the base64 values) and no stringData. The stringData field is write-only convenience: it is never returned.

Never commit a Secret YAML file to Git. Even a base64 one. A repository is read by many people and it keeps history forever. The file above is only safe in this lesson because the password is made up.

A leak most people do not know about: kubectl apply

When I ran kubectl get secret notes-db -o yaml after kubectl apply, I saw this:

YAML
metadata:  annotations:    kubectl.kubernetes.io/last-applied-configuration: |      {"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{},"name":"notes-db","namespace":"default"},"stringData":{"DB_PASSWORD":"s3cr3t-pw","DB_USER":"notes"},"type":"Opaque"}

That is a plain text copy of my password in an annotation. kubectl apply stores the last thing you applied, so it can work out later what changed. If you apply a Secret that uses stringData, the plain value is saved there. People with permission to read the Secret can read it anyway, so it is not a new door, but it shows up in logs, kubectl get -o yaml pastes and backups. It is another reason not to manage Secrets by hand-written files. Create them with kubectl create secret (I checked: a Secret made this way had no annotations at all), or use an external secret tool, or use server-side apply.

Where does a Secret really live? I looked in etcd

The API server keeps every object in etcd. I have a real etcd in my lab, so I read the Secret straight out of it with etcdctl (a tool for reading etcd). By default, Kubernetes stores Secrets in etcd without encryption:

Shell
etcdctl get /registry/secrets/default/notes-db --print-value-only | strings | grep -c s3cr3t
TEXT
2

Two lines of the stored bytes contain my password in plain text: one in the last-applied-configuration annotation and one in the data. Anyone with access to the etcd disk, or an etcd backup, could read all the Secrets in the cluster.

Now I did what the Kubernetes docs describe. I wrote an EncryptionConfiguration file and restarted the API server with --encryption-provider-config. This is the file (with the key removed):

YAML
# encryption-config.example.yamlapiVersion: apiserver.config.k8s.io/v1kind: EncryptionConfigurationresources:  - resources:      - secrets    providers:      - aescbc:          keys:            - name: key1              secret: <base64 of 32 random bytes>      - identity: {}

The first provider in the list is used to write. All providers listed are tried when reading. identity: {} means "no encryption" and it comes last so that older, unencrypted Secrets can still be read. After restarting the API server, I created a new Secret and looked at its bytes:

TEXT
k8s:enc:aescbc:v1:key1:R

The stored value starts with k8s:enc:aescbc:v1:key1:, which says "encrypted with AES-CBC using key1", and the rest is unreadable. A search for the password in the stored bytes now finds 0 lines. The API still returns the real value to a caller who is allowed to read it:

Shell
kubectl get secret fresh -o jsonpath='{.data.DB_PASSWORD}' | base64 -d
TEXT
s3cr3t-pw

The old Secret was still plain in etcd until I rewrote it. The docs suggest rewriting all Secrets after turning on encryption. I did that with:

Shell
kubectl get secrets -A -o json | kubectl replace -f -

and then the old notes-db Secret also started with k8s:enc:aescbc:v1:key1: and had 0 matches for the password.

Two honest notes. First, I used aescbc with the key in a file on the API server, because it is the simplest to show. The docs say a KMS provider (a key held in a cloud key service or hardware, outside the cluster) is stronger, because the key is not next to the data. Managed services like EKS, GKE and AKS have their own switch for this. Second, encryption at rest protects the disk and backups. It does nothing about someone who has legitimate API access, which brings us to RBAC.

Who can read Secrets: RBAC

RBAC (role-based access control) says which user or service account may do which action on which object. The rule to remember: get, list and watch on secrets all reveal the values. Even list does, because it returns the full objects. Also, anyone who can create a pod in a namespace can mount any Secret from that namespace into it and print it. So giving someone "create pods" is almost the same as giving "read all secrets here".

To check what an account may do, use kubectl auth can-i. In my lab I ran it as admin and as the default service account of a namespace:

TEXT
$ kubectl -n lab8 auth can-i get secretsyes$ kubectl -n lab8 auth can-i get secrets --as=system:serviceaccount:lab8:defaultno

A fresh service account has no permissions by default. Keep it that way for your apps.

Secrets in env vars or in files?

Both work (secretKeyRef / secretRef for env, or a secret volume). Files are a little safer, for three reasons. Environment variables are easy to leak: crash reporters, printenv in a debug session, /proc/<pid>/environ, and child processes all see them. A file can have strict permissions. And volume-mounted Secrets are updated eventually when they change, while env vars never are. The trade-off is that many apps expect env vars. If you use env vars, make sure your app never logs them.

Other kinds of Secret

A Secret has a type. Opaque is the default ("any data"). Others have required keys, which the API checks. The ones you meet most:

TypeUsed for
OpaqueYour own values (default)
kubernetes.io/dockerconfigjsonCredentials to pull images from a private registry
kubernetes.io/tlsA certificate and key (tls.crt, tls.key), used by Ingress
kubernetes.io/basic-auth, ssh-authBasic auth and SSH keys

The registry one deserves an example, because the ImagePullBackOff problem in lesson 8 often comes from it. This creates the Secret:

Shell
kubectl create secret docker-registry regcred \  --docker-server=registry.example.com \  --docker-username=bot --docker-password=pw \  --dry-run=client -o yaml
TEXT
apiVersion: v1data:  .dockerconfigjson: eyJhdXRocyI6eyJyZWdpc3RyeS5leGFtcGxlLmNvbSI6eyJ1c2VybmFtZSI6ImJvdCIsInBhc3N3b3JkIjoicHciLCJhdXRoIjoiWW05ME9uQjMifX19kind: Secretmetadata:  name: regcredtype: kubernetes.io/dockerconfigjson

Then the pod says imagePullSecrets: [{name: regcred}] so the kubelet can log in when it pulls the image. Like all Secrets, it must be in the same namespace as the pod.

Better places for secrets

For real production use, many teams keep the source of truth outside the cluster (a cloud secret manager or a vault) and sync it in with a tool such as the External Secrets Operator or a CSI driver. Others keep encrypted Secret files in Git (Sealed Secrets, SOPS). I mention the names so you can recognise them. I did not run any of these here.

Volumes: giving a pod a folder

A container's own filesystem is thrown away when the container is replaced. Lesson 2 showed this with Docker. In Kubernetes a volume is a folder that belongs to the pod and can be mounted into one or more containers of that pod. The ConfigMap volume above is one kind. Here are the kinds you need first.

VolumeLifetimeTypical use
emptyDirSame as the pod. Deleted when the pod goes. It survives a container restart inside the same pod.Scratch space, cache, sharing files between two containers in one pod
configMap, secretFollows the objectConfig files, certificates
persistentVolumeClaimOutlives the podDatabases, uploads
hostPathThe node's folderSpecial system pods only. Avoid for apps.

The hard question is the third row: how do you get a disk that outlives pods? Kubernetes splits that job in two, so that app developers do not need to know which cloud they run on.

PersistentVolumes and claims: the two-sided deal

Think of renting an apartment. The PersistentVolume (PV) is the apartment: a real piece of storage, such as a cloud disk or an NFS share. The PersistentVolumeClaim (PVC) is your rental request: "I need 5 GiB, I will write to it". The cluster matches the request to an apartment. Your pod then says "mount that claim".

The benefit: the developer writes the claim and never needs to know if the disk is an AWS EBS volume, a Google Persistent Disk, or a local SSD.

Static setup (an admin creates PVs by hand) is rare today. Most clusters use dynamic provisioning: you create the claim, and a StorageClass makes the PV for you.

A claim and a pod that uses it:

YAML
# storage.yamlapiVersion: v1kind: PersistentVolumeClaimmetadata:  name: uploadsspec:  accessModes:    - ReadWriteOnce  resources:    requests:      storage: 5Gi  # storageClassName omitted: the cluster default StorageClass is used---apiVersion: v1kind: Podmetadata:  name: uploaderspec:  containers:    - name: app      image: notes-api:1.0      volumeMounts:        - name: data          mountPath: /var/lib/notes/uploads  volumes:    - name: data      persistentVolumeClaim:        claimName: uploads

Status: accepted by the real v1.34.0 API server (and passed kubeconform -strict). I will show what the control plane really did with it in a moment. A real attach and mount needs a node and a storage provisioner, so that part is reviewed, not run on a cluster here.

If you omit storageClassName, the cluster's default StorageClass is used. If the cluster has no default and there is no matching PV, the claim stays Pending. That is exactly what happened in my lab, which has no StorageClass at all. This is what a real PVC in that state looks like:

TEXT
$ kubectl get pvc uploadsNAME      STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGEuploads   Pending                                                     <unset>                 0s

and the event the control plane recorded:

TEXT
Normal  FailedBinding  persistentvolumeclaim/uploads  no persistent volumes available for this claim and no storage class is set

(The event type is Normal, even though the message sounds bad. It means "I tried to bind and could not yet".) The pod that uses this claim shows that it is waiting too. The scheduler said, in the real run:

TEXT
Warning  FailedScheduling  default-scheduler  0/1 nodes are available: persistentvolumeclaim "no-such-claim" not found. not found

That one was my deliberate test with a claim name that does not exist. We will use it in lesson 8. For now remember: a pod with a PVC that cannot bind will stay Pending.

StorageClass

A StorageClass says how to make storage. Here is one I validated:

YAML
apiVersion: storage.k8s.io/v1kind: StorageClassmetadata:  name: fastprovisioner: example.com/fast-diskreclaimPolicy: RetainvolumeBindingMode: WaitForFirstConsumerallowVolumeExpansion: true

(example.com/fast-disk is a made-up provisioner. A real cluster uses the name of its CSI driver here. Reviewed, not run on a cluster here, other than the API server accepting it.)

Three fields matter. I read each in kubectl explain on the real server:

  • volumeBindingMode: Immediate (the default) creates the disk as soon as the claim exists. WaitForFirstConsumer waits until a pod that uses the claim is scheduled, and then creates the disk in the right place. This matters because a cloud disk lives in one zone. If the disk is made in zone A and the pod lands in zone B, the pod can never start. WaitForFirstConsumer avoids that.
  • reclaimPolicy: what happens to the disk when the claim is deleted. Delete (the default for dynamically provisioned volumes) deletes the real disk and your data. Retain keeps it, for you to clean up by hand. For anything precious, think about this before you delete a claim.
  • allowVolumeExpansion: whether you may make a claim bigger later by editing the size. (You can grow a volume, not shrink it.)

Access modes

ModeShortMeaning
ReadWriteOnceRWORead-write by one node at a time. Many pods on that same node may still use it.
ReadWriteOncePodRWOPRead-write by exactly one pod.
ReadOnlyManyROXRead-only by many nodes.
ReadWriteManyRWXRead-write by many nodes. Needs shared storage such as NFS or a cloud file service.

The trap: ordinary cloud block disks are RWO. If you scale a Deployment with a RWO claim to three replicas, pods that land on other nodes cannot mount the disk and stay stuck. One shared claim in a multi-replica Deployment is therefore a design smell. Use a StatefulSet (next), or move the shared data to object storage or a database.

Deleting things in the right order

Kubernetes has "storage object in use protection". If you delete a PVC that a pod still uses, the PVC is marked for deletion but is kept until the pod is gone. And deleting a PVC with reclaimPolicy: Delete destroys the data. Always know which policy applies before you run kubectl delete pvc.

StatefulSets: pods with names and their own disks

A Deployment treats pods as identical cattle. Pod notes-api-7888f544c5-hsp6l is replaceable by any other. A database is different. Postgres member 0 has data. Member 1 has its own copy. Each needs its own disk, and others need to find "member 0" by name.

A StatefulSet gives each pod three things:

  1. A stable name with a number: pg-0, pg-1, pg-2 (the number is called the ordinal). When pg-1 is replaced, the new pod is again called pg-1.
  2. A stable network name through a headless Service (a Service with clusterIP: None, from lesson 5). Each pod gets a DNS name like pg-0.pg.
  3. Its own storage, made from volumeClaimTemplates. Each pod gets its own PVC (data-pg-0, data-pg-1, ...), and when the pod is replaced the same claim is attached to the new pod.

By default the pods start in order (0, then 1, then 2) and stop in the reverse order.

YAML
# statefulset.yamlapiVersion: v1kind: Servicemetadata:  name: pgspec:  clusterIP: None  selector:    app: pg  ports:    - name: postgres      port: 5432---apiVersion: apps/v1kind: StatefulSetmetadata:  name: pgspec:  serviceName: pg  replicas: 3  selector:    matchLabels:      app: pg  template:    metadata:      labels:        app: pg    spec:      containers:        - name: postgres          image: postgres:17          ports:            - name: postgres              containerPort: 5432          env:            - name: POSTGRES_PASSWORD              valueFrom:                secretKeyRef:                  name: notes-db                  key: DB_PASSWORD            - name: PGDATA              value: /var/lib/postgresql/data/pgdata          volumeMounts:            - name: data              mountPath: /var/lib/postgresql/data  volumeClaimTemplates:    - metadata:        name: data      spec:        accessModes:          - ReadWriteOnce        resources:          requests:            storage: 10Gi

(PGDATA points to a sub-folder because a fresh volume often has a lost+found folder at its root, and Postgres refuses to start in a non-empty folder. That is advice from the Postgres image docs, not something I could test here.)

Status: applied to the real v1.34.0 API server. kubectl get statefulset pg then showed READY 0/3, and the controller recorded these real events:

create Claim data-pg-0 Pod pg-0 in StatefulSet pg success create Pod pg-0 in StatefulSet pg successful

The first pod and its claim were made. Pod pg-0 stayed Pending, because there was no node to run it and no storage to bind. The API server also printed a warning, spec.SessionAffinity is ignored for headless services, which is a harmless note from the API's defaulting. Everything after "the pod starts" is reviewed, not run on a cluster here.

Notice the sentence "create Claim data-pg-0": the claim name is <template name>-<pod name>. That is what you look for when you debug storage.

What a StatefulSet does not do

A StatefulSet gives stable identity and storage. It does not set up database replication, handle backups, or do failover. Postgres still needs to be told how to be a primary and replicas. That is why people use an operator (a program that knows how to run one particular database) rather than writing StatefulSets by hand.

Also, the PVCs of a StatefulSet are kept when you scale down or delete the StatefulSet. That is safe by default (you do not lose data by accident), but it leaves disks that you pay for. The field persistentVolumeClaimRetentionPolicy has whenDeleted and whenScaled, each Retain (default) or Delete. I confirmed that in kubectl explain on the real server.

My honest opinion for a beginner: in production, a managed database (a cloud service) is usually safer than your own Postgres in a StatefulSet. Run databases in Kubernetes when you have a good reason and an operator you trust.

Commands to know

CommandWhat it does
kubectl create configmap NAME --from-literal=K=V --from-file=FILEMake a ConfigMap
kubectl create secret generic NAME --from-literal=K=VMake a Secret
kubectl get configmap NAME -o yamlSee the stored data
kubectl get secret NAME -o jsonpath='{.data.KEY}' | base64 -dRead one Secret value
kubectl rollout restart deployment/NAMEReplace pods so they re-read env vars
kubectl get pvc, kubectl get pv, kubectl get storageclassSee storage objects
kubectl describe pvc NAMESee binding events
kubectl auth can-i get secrets --as=USERTest permissions

A tip I learned while testing. kubectl apply --dry-run=client needs a reachable API server even for client-side checking, because kubectl asks the server for its list of resource types. With no cluster, my run printed error validating ... failed to download openapi ... connection refused, and with --validate=false it printed unable to recognize ... connection refused. And kubectl explain printed The connection to the server localhost:8080 was refused. Only generators like kubectl create configmap ... --dry-run=client -o yaml work with no server. So with no cluster use kubeconform and yamllint. And kubeconform has a trap: in non-strict mode it accepted my Deployment with the typo envfrom (wrong case), because unknown fields are allowed by default. With -strict it said additional properties 'envfrom' not allowed. The real API server also rejected it: strict decoding error: unknown field "spec.template.spec.containers[0].envfrom". Always use -strict.

Try it

You need kubectl and, ideally, kubeconform and a local cluster (kind, minikube or Docker Desktop). Without a cluster, steps 1 to 3 still work.

  1. Create a ConfigMap with kubectl create configmap app --from-literal=MODE=prod --dry-run=client -o yaml. Then make the same ConfigMap with PAGE_SIZE: 20 (no quotes) in a file and run kubeconform -strict on it. What does it say?
  2. Make a Secret with --from-literal=TOKEN=abc123 --dry-run=client -o yaml. Decode the value without any Kubernetes tool.
  3. Write a Deployment that reads LOG_LEVEL from a ConfigMap as an environment variable. Run kubeconform -strict. Now change the key name configMapKeyRef to configMapKeyref and run it again.
  4. On a real cluster: change the ConfigMap value after the pod started. Does printenv LOG_LEVEL in the pod change? Does a mounted file change? What do you do to make the env var update?
  5. On a real cluster: create a PVC without a storage class. Is it Pending or Bound? What decides this?
  6. Why would you choose WaitForFirstConsumer for a cloud cluster that spans three zones?

Answers

  1. PAGE_SIZE: 20 fails with got number, want null or string. Quote it: "20".
  2. echo -n 'YWJjMTIz' | base64 -d prints abc123. (Use -n when encoding so that no newline is added into the value: echo -n abc123 | base64.) The point: base64 is reversible by anyone.
  3. The second run fails with additional properties 'configMapKeyref' not allowed (strict mode catches the typo).
  4. The env var does not change in the running pod. A mounted file does change after a delay (unless you used subPath). To update the env var, restart the pod: kubectl rollout restart deployment/NAME.
  5. It depends on whether the cluster has a default StorageClass. With one, a volume is made and (with Immediate binding) the claim becomes Bound, or it waits for a pod if the class uses WaitForFirstConsumer. With none, it stays Pending with no persistent volumes available for this claim and no storage class is set.
  6. A cloud disk exists in one zone. WaitForFirstConsumer creates it after the scheduler picks a node, so the disk is in the same zone as the pod.

Interview questions

What is the difference between a ConfigMap and a Secret? (Must know)

Both are objects with key and value data that pods can use as environment variables or files. A ConfigMap is for normal settings. A Secret is for sensitive values. The big point I make is that a Secret is not encrypted by default: its values are only base64, and by default they are stored in etcd as they are. What makes a Secret safer is the surrounding setup: RBAC so only a few accounts can read it, encryption at rest in etcd, and the kubelet keeping secret volumes in memory. If I need strong secrecy I also use an external secret manager.

Is base64 encryption? (Trap)

No. It is an encoding that anyone can reverse with one command. I can show it: echo 'czNjcjN0LXB3' | base64 -d prints the password. It exists so binary data can sit in YAML and JSON. The trap is when a candidate says Secrets are "encrypted with base64".

How do you make Secrets really protected? (Senior signal)

I work on three levels. First, access: tight RBAC, remembering that list and get show values and that "create pods" lets someone mount any Secret in the namespace. Second, at rest: turn on encryption for secrets in the API server, ideally with a KMS provider. I have tried an EncryptionConfiguration with aescbc in a lab and saw the stored value change from readable to k8s:enc:aescbc:v1:.... I also know existing Secrets must be rewritten after enabling it. Third, source: keep them out of Git and out of images, and use an external secret manager or sealed secrets. I also avoid env vars for the most sensitive values because they leak into logs and debug sessions.

You change a ConfigMap, but the app still uses the old value. Why? (Trap)

It depends on how the pod uses the ConfigMap. Environment variables are set once when the container starts, so a running container never sees the change; I need a new pod, for example with kubectl rollout restart. A volume-mounted ConfigMap updates after a delay (the kubelet sync period plus cache time), but the app must re-read the file, and a subPath mount never updates. A good fix is to put a hash of the config in the pod template, or use immutable ConfigMaps with a versioned name, so a config change is a normal rolling update that can be rolled back.

What is an immutable ConfigMap or Secret and why use it? (Senior signal)

It is one with immutable: true. After creation its data cannot be changed, and the API server rejects the edit (I got field is immutable when immutable is set). To change config, I make a new object with a new name and point the Deployment at it. This protects running pods from a bad live edit, makes rollbacks easy, and lowers load on the API server because the kubelet does not need to watch it.

Explain PV, PVC and StorageClass. (Must know)

A PersistentVolume is a real piece of storage in the cluster. A PersistentVolumeClaim is a request for storage by a user, with size and access mode. A StorageClass describes how to create storage on demand. With dynamic provisioning I create the PVC, the StorageClass provisioner makes a PV, and Kubernetes binds them. The pod only refers to the claim, so the app does not care which cloud disk is behind it.

A PVC is stuck in Pending. What do you check? (Trap)

I run kubectl describe pvc and read the events. Common causes: there is no default StorageClass and the claim names none, so the event says no persistent volumes available for this claim and no storage class is set (I saw this exact event); the named StorageClass does not exist; the provisioner is not installed or is failing; the class uses WaitForFirstConsumer, so the claim is supposed to stay Pending until a pod uses it; or no PV matches the size or access mode in a static setup. Pending with WaitForFirstConsumer is normal, which is the trap.

What do the access modes mean, and what goes wrong with ReadWriteOnce? (Senior signal)

ReadWriteOnce allows read-write from one node. ReadWriteOncePod allows one pod. ReadOnlyMany and ReadWriteMany allow many nodes, and RWX needs shared storage like NFS or a managed file service. The common mistake is a Deployment with several replicas that share one RWO claim. Pods on a different node cannot attach the disk and stay stuck. The fix is a StatefulSet with one claim per pod, shared storage if the app truly needs it, or moving the data to a database or object storage.

Deployment or StatefulSet for a database, and what does a StatefulSet really give you? (Senior signal)

A StatefulSet. It gives each pod a stable name (pg-0), a stable DNS name through a headless Service, and its own PVC from volumeClaimTemplates that follows the pod across restarts, with ordered start and stop. It does not give replication, backups or failover; the database software or an operator does that. Also its PVCs are kept on scale-down or delete by default, which protects data but costs money, and I can change that with persistentVolumeClaimRetentionPolicy. For production I often prefer a managed database.

What does reclaimPolicy do? (Trap)

It decides what happens to the real disk when its claim is deleted. Delete, the default for dynamically provisioned volumes, deletes the PV and the underlying disk, so the data is gone. Retain keeps the disk for a person to clean up. The trap is deleting a PVC in a hurry on a Delete class. For important data I use Retain or backups and check the policy first.

Why is kubectl apply a risk for Secrets? (Senior signal)

kubectl apply stores the manifest you applied in the annotation kubectl.kubernetes.io/last-applied-configuration. If the manifest used stringData, my real run showed the plain password inside that annotation. It is visible to anyone who can read the Secret, and it ends up in YAML pastes and backups. I avoid it by creating Secrets with kubectl create secret or through an external tool, and never committing Secret manifests.

Key takeaways

  • Keep settings out of the image. ConfigMaps hold normal settings, Secrets hold sensitive ones, and the pod reads them as env vars or files.
  • ConfigMap and Secret values are strings. Quote numbers and booleans. kubeconform -strict and the API server both catch this.
  • Env vars from a ConfigMap are fixed at container start. Volume files update later (not with subPath). Use rollout restart, a config hash, or immutable versioned ConfigMaps.
  • base64 is not encryption. By default Secrets sit in etcd in plain form (I read one there). Use RBAC, encryption at rest (best with KMS), and an external secret source.
  • kubectl apply can leave a plain copy of a Secret in an annotation.
  • PVC asks for storage, PV is the storage, StorageClass makes it on demand. A pod that needs an unbound PVC stays Pending.
  • Use WaitForFirstConsumer for zone-bound cloud disks, and check reclaimPolicy before deleting a claim.
  • StatefulSets give stable names, stable DNS and a disk per pod. They do not give you database replication.
  • Always run kubeconform -strict. Non-strict mode misses misspelled field names.

What's next

Next: Scaling, Rollouts and Reliability: Probes, Resources and Autoscaling, Lesson 7. Your pods now have settings, secrets and disks. Next they must stay healthy while you update and scale them. You will meet readiness and liveness probes, requests and limits, rolling updates and rollbacks, the autoscaler and PodDisruptionBudgets. Practise now: add a ConfigMap and a Secret to one of your own Deployments, and check it with kubeconform -strict.

Published by Jainik · Last updated · How these lessons are made

Keep learning

Related lessons

Part of the study guide

Related series