Full-Stack Questions

Containers, Kubernetes, Terraform and AWS for Full-Stack Interviews

Containers versus VMs, Kubernetes objects, probes, rollouts, Helm, platform choice, IAM, Terraform state, secrets and cost.

Lesson 3438 min read#kubernetes#terraform#aws

On this page

  1. Words used in this lesson
  2. Containers vs VMs
  3. What Kubernetes does, and the objects you must know
  4. Probes: how Kubernetes knows your app is fine
  5. Requests and limits
  6. Rolling updates and rollbacks
  7. Helm basics
  8. Serverless vs containers vs PaaS: how to choose
  9. Core AWS services for a full-stack developer
  10. Terraform basics
  11. Secrets
  12. Cost awareness
  13. Multi-environment strategy
  14. Common mistakes
  15. Interview questions
  16. Key takeaways
  17. What's next

In one line: a container packages your app so it runs the same everywhere, Kubernetes keeps a fixed number of healthy copies running and replaces them safely, and Terraform writes your cloud setup as code you can review. A strong answer picks the simplest platform that fits (PaaS, serverless, ECS, then Kubernetes), uses least-privilege access, keeps secrets out of code, and watches the bill.

At 4+ years you are not expected to run a cluster alone. You are expected to read a Deployment file, explain why a pod keeps restarting, choose between Lambda and a container, write a safe IAM policy, and say what terraform plan is telling you. This lesson gives the words and the model answers.

It does not repeat the hands-on steps. For Docker, read DevOps, Lesson 9. For IAM basics, read Lesson 5. For VPC, Lesson 6. For RDS, Lesson 8. For S3 and CloudFront, Lesson 11. For load balancers and auto scaling, Lesson 12. For Terraform step by step, Lesson 15. For cost, Lesson 19. The release side (pipelines, rollbacks, Sentry, SLOs) is in Deployment, CI/CD and Monitoring. Here we look at what is asked in the interview room.

What I ran. I validated every Kubernetes manifest here with kubeconform in strict mode against the Kubernetes 1.34 schema (and re-ran that check while reviewing this lesson; 1.34 is the release where the preStop sleep action became stable). Newer Kubernetes releases exist, so check the version your cluster runs. I ran the Terraform samples with Terraform 1.16 (using built-in resources, because the AWS provider could not be downloaded in my sandbox, so the AWS snippets were format-checked only). I ran the Node health server and compiled the Lambda handler with TypeScript. I could not run Helm, so I only state Helm facts I checked in its source.

Words used in this lesson

  • A container is a running process that is isolated from others (its own files, network and process list) but shares the host's operating system kernel. Think of a flat in a building: separate rooms, shared pipes.
  • A virtual machine (VM) is a whole fake computer with its own operating system, running on real hardware. Think of a separate house.
  • Kubernetes (K8s) is a system that runs containers across many machines and keeps them in the state you asked for. You say "I want 3 copies of this image", and it makes that true and keeps it true.
  • A cluster is a set of machines (nodes) run by Kubernetes. The control plane is the part that makes decisions. Nodes run your containers.
  • Declarative means you describe the result you want, not the steps. Reconciliation is the loop where Kubernetes compares "what you asked for" with "what exists" and fixes the difference.
  • A manifest is a YAML file that describes one Kubernetes object.
  • Infrastructure as Code (IaC) means you describe servers, databases and networks in files, store them in Git, and apply them with a tool, instead of clicking in a console.
  • Serverless means you give the cloud a function or a container and it runs it on demand, scales it, and bills by use. There are still servers; you just do not manage them.
  • PaaS (Platform as a Service) is a host that builds and runs your app from your code with almost no setup, such as Vercel, Render or Heroku.
  • Least privilege means giving a person or program only the permissions it needs for its job, and nothing more.

1. Containers vs VMs

Virtual machineContainer
IsolationStrong: own kernelWeaker: shares the host kernel
Start timeTens of seconds to minutesUsually a second or less
SizeGigabytesTens to hundreds of megabytes
What you shipA machine imageAn app image (see Docker, Lesson 9)
Good forDifferent operating systems, strong tenant isolationPacking many app copies on one machine, fast deploys

Containers are not "lighter VMs". They are normal processes with walls built from Linux features (namespaces limit what a process can see, cgroups limit how much CPU and memory it can use). That is why a container cannot run a different kernel, and why a kernel bug can matter to every container on the host.

Difference in one line: a VM virtualizes the hardware and runs a full operating system; a container virtualizes the operating system and runs one app.

Most of the cloud is both: your containers run on VMs (EC2 nodes), or on a managed layer that hides the VMs (Fargate).

2. What Kubernetes does, and the objects you must know

You do not tell Kubernetes "start container X on machine 3". You send it an object that says what you want. Controllers inside Kubernetes watch for differences and fix them. If a node dies, the Deployment controller notices that only 2 of 3 pods exist and creates another.

The objects, in order of use

  • A Pod is the smallest thing Kubernetes runs: one or more containers that share a network address and storage. Usually one container per pod. Pods are disposable: they get a new IP when replaced.
  • A Deployment is an object that keeps N identical pods running and handles updates. It creates a ReplicaSet (one per version) that does the counting. You almost never create bare pods.
  • A Service is a stable name and IP in front of a changing set of pods. It finds pods by labels (key-value tags). Types: ClusterIP (inside the cluster, the default), NodePort, and LoadBalancer (asks the cloud for a load balancer).
  • An Ingress is a set of HTTP rules (host and path to Service) that an ingress controller (a program such as NGINX or a cloud load balancer controller) turns into real routing. The Gateway API is the newer, more expressive replacement; the Kubernetes docs say the Ingress API is frozen (still supported, no new features) and recommend Gateway for new work. Also note: the Kubernetes project announced the retirement of the community ingress-nginx controller, with best-effort maintenance only until March 2026 (see its README). The nginx class in my example is only a placeholder; for new work pick a controller that is still maintained, or use a cloud load balancer controller or a Gateway API implementation.
  • A ConfigMap holds non-secret settings. A Secret holds sensitive values (passwords, tokens).
  • An HPA (HorizontalPodAutoscaler) changes the number of replicas of a Deployment based on a metric such as CPU.

Difference in one line: a Pod runs the app, a Deployment keeps pods alive and updates them, a Service gives them one stable address.

One complete example

This file creates all of them for a Node API. It passed kubeconform --strict against Kubernetes 1.34.

YAML
apiVersion: v1kind: ConfigMapmetadata:  name: shop-api-configdata:  LOG_LEVEL: info  FEATURE_NEW_CHECKOUT: "false"---apiVersion: v1kind: Secretmetadata:  name: shop-api-secretstype: OpaquestringData:  DATABASE_URL: postgres://shop:CHANGE_ME@db.internal:5432/shop---apiVersion: apps/v1kind: Deploymentmetadata:  name: shop-api  labels:    app: shop-apispec:  replicas: 3  revisionHistoryLimit: 5  selector:    matchLabels:      app: shop-api  strategy:    type: RollingUpdate    rollingUpdate:      maxSurge: 1      maxUnavailable: 0  template:    metadata:      labels:        app: shop-api    spec:      terminationGracePeriodSeconds: 30      containers:        - name: api          image: registry.example.com/shop-api:1.4.2          ports:            - containerPort: 3000          envFrom:            - configMapRef:                name: shop-api-config            - secretRef:                name: shop-api-secrets          resources:            requests:              cpu: 250m              memory: 256Mi            limits:              memory: 256Mi          startupProbe:            httpGet:              path: /health/live              port: 3000            periodSeconds: 5            failureThreshold: 24          livenessProbe:            httpGet:              path: /health/live              port: 3000            periodSeconds: 10            failureThreshold: 3          readinessProbe:            httpGet:              path: /health/ready              port: 3000            periodSeconds: 5            failureThreshold: 2          lifecycle:            preStop:              sleep:                seconds: 5---apiVersion: v1kind: Servicemetadata:  name: shop-apispec:  selector:    app: shop-api  ports:    - port: 80      targetPort: 3000---apiVersion: networking.k8s.io/v1kind: Ingressmetadata:  name: shop-apispec:  ingressClassName: nginx  tls:    - hosts: [api.example.com]      secretName: api-example-com-tls  rules:    - host: api.example.com      http:        paths:          - path: /            pathType: Prefix            backend:              service:                name: shop-api                port:                  number: 80---apiVersion: autoscaling/v2kind: HorizontalPodAutoscalermetadata:  name: shop-apispec:  scaleTargetRef:    apiVersion: apps/v1    kind: Deployment    name: shop-api  minReplicas: 3  maxReplicas: 10  metrics:    - type: Resource      resource:        name: cpu        target:          type: Utilization          averageUtilization: 70  behavior:    scaleDown:      stabilizationWindowSeconds: 300

Read it from the top. The Deployment's selector must match the pod template labels, and the Service's selector matches the same labels. That label match is how everything finds everything.

Never put a real password in a file that goes to Git, even in a Secret. The file above uses CHANGE_ME on purpose. Section 10 shows what to do instead.

A good habit: run kubeconform -strict (or kubectl apply --dry-run=server) in CI. With -strict, a typo like replica: 3 is rejected as an unknown field. I ran this check and it failed as expected. Note that kubectl --dry-run=client still wants to talk to an API server for the object list, so it is not a pure offline linter.

3. Probes: how Kubernetes knows your app is fine

A probe is a check that Kubernetes runs against a container on a timer. There are three kinds, and the answer to "which one" is a classic interview question.

ProbeQuestion it asksIf it fails
StartupHas the app finished starting?Keep waiting; after too many failures, restart. Liveness and readiness are paused until it passes.
LivenessIs the process stuck for good?The container is restarted
ReadinessCan it take traffic right now?The pod is removed from the Service (no restart)

In my example the startup probe allows 5 s x 24 = 120 s to boot. After that, liveness takes over with a tight setting.

The most common mistake: putting a database check in the liveness probe. If the database is slow, every pod fails liveness, Kubernetes restarts them all, and now the app is down and restarting, which also hits the database harder when they come back. Liveness should only say "this process is broken". A slow dependency should make the pod not ready, not dead. (Same point as in Lesson 26.)

A tiny Node server with both endpoints and a graceful shutdown. I ran it and sent SIGTERM; it exited with code 0.

JavaScript
import http from "node:http";let ready = false;let shuttingDown = false;const server = http.createServer((req, res) => {  if (req.url === "/health/live") {    res.writeHead(200).end("ok"); // process is alive; do NOT check the database here  } else if (req.url === "/health/ready") {    const ok = ready && !shuttingDown; // add a cheap dependency check here if needed    res.writeHead(ok ? 200 : 503).end(ok ? "ready" : "not ready");  } else {    res.writeHead(200).end("hello");  }});server.listen(3000, () => {  ready = true;});process.on("SIGTERM", () => {  shuttingDown = true; // readiness now fails, so the Service stops sending new traffic  server.close(() => process.exit(0)); // finish in-flight requests, then exit  setTimeout(() => process.exit(1), 25_000).unref(); // stay below terminationGracePeriodSeconds});

Why preStop sleep? When a pod is deleted, two things start at about the same time: Kubernetes sends SIGTERM to your container, and it begins removing the pod from the Service. The second part spreads through the cluster and takes a moment, so for a short time new requests can still arrive at a pod that is shutting down. A few seconds of preStop sleep (before SIGTERM) hides that gap. The default grace period is 30 seconds, after which the container is killed. More on graceful shutdown: Backend, Lesson 4.

4. Requests and limits

Each container can set two numbers per resource:

  • A request is the amount the scheduler reserves for the container. The scheduler places the pod on a node that has this much free. It is also the base for the HPA's CPU percentage.
  • A limit is the maximum the container may use.

CPU is measured in cores (250m = a quarter of a core). Memory is in bytes (256Mi).

What happens at the limit is different for each resource, and interviewers love this:

ResourceOver the limitResult
CPUThe container is throttled (slowed)Higher latency, no crash
MemoryThe container is killedOOMKilled, exit code 137, then restart

CPU can be squeezed and shared, memory cannot. QoS (quality of service) classes decide who is evicted first when a node runs short: BestEffort (no requests or limits) goes first, then Burstable, and Guaranteed (requests equal limits for every resource) last.

My common setup: memory request equals memory limit, CPU request set, no CPU limit. The reasoning is that a CPU limit can throttle a Node app even when the node has idle CPU, which shows up as odd latency spikes. This is a common practice, not a rule, and some platforms or teams require CPU limits for fairness. State your trade-off out loud.

Node.js and memory limits: if the container has a 256Mi limit but your heap can grow past it, the kernel kills the process with no stack trace. Watch OOMKilled in kubectl describe pod, set --max-old-space-size below the limit, and leave room for buffers and native memory.

Without any requests, the HPA cannot calculate "70% of requested CPU", and the scheduler cannot place pods well. Always set requests.

5. Rolling updates and rollbacks

A Deployment has a strategy. The default is RollingUpdate: it creates new pods and removes old ones step by step, so there is no downtime if the pods have good readiness probes.

Two numbers control the speed:

  • maxSurge is how many extra pods above the desired count may exist during the update.
  • maxUnavailable is how many pods may be missing during the update.

Both default to 25%. My example uses maxSurge: 1, maxUnavailable: 0: start one new pod, wait until it is ready, then remove one old pod. It is slower but never lowers capacity. Without a readiness probe, "ready" means "container started", and the rollout can send traffic to a pod that is still booting. The readiness probe is the safety belt of rolling updates.

The other strategy is Recreate (stop all old, then start new). Use it only when two versions must never run together.

Shell
kubectl set image deploy/shop-api api=registry.example.com/shop-api:1.4.3kubectl rollout status deploy/shop-api     # waits; non-zero exit if it failskubectl rollout history deploy/shop-apikubectl rollout undo deploy/shop-api       # back to the previous revisionkubectl rollout undo deploy/shop-api --to-revision=2

If new pods never become ready, the rollout stalls. Because maxUnavailable is 0 here, the old pods keep serving, so users see no outage. After progressDeadlineSeconds (default 600 s) the Deployment is marked as failed (ProgressDeadlineExceeded) and kubectl rollout status returns an error, which is how CI notices. It does not roll back by itself. revisionHistoryLimit (default 10) is how many old ReplicaSets are kept for undo.

Rolling back the image does not roll back the database. If version 1.4.3 ran a migration that dropped a column, undo brings back code that needs the column. Use expand-then-contract migrations, as in Lesson 26.

Other rollout facts: use an exact image tag (or digest) and never :latest, otherwise two pods can run different code under the same name. Changing a ConfigMap does not restart pods that read it as environment variables; run kubectl rollout restart deploy/shop-api or change an annotation. For pods that must survive node maintenance, add a PodDisruptionBudget (a rule such as "at least 2 pods must stay up during voluntary evictions").

Debugging a pod that does not work

What you seeUsual causeFirst command
PendingNo node has enough free CPU or memory for the requestskubectl describe pod (look at Events)
ImagePullBackOffWrong tag, or no pull credentialskubectl describe pod
CrashLoopBackOffApp exits on start (bad env, bad config)kubectl logs --previous
OOMKilledMemory above the limitkubectl describe pod (Last State)
Running but 0/1 readyReadiness probe failingkubectl describe pod, then the probe path
Service gives no responseSelector does not match pod labelskubectl get endpoints shop-api

6. Helm basics

Helm is a package manager for Kubernetes. A chart is a folder of YAML templates plus a values.yaml file with default settings. A release is one installed copy of a chart in a cluster, with its own history. The template fills the blanks from values:

YAML
# values.yamlimage:  repository: registry.example.com/shop-api  tag: "1.4.2"replicaCount: 3
YAML
# templates/deployment.yaml (fragment)spec:  replicas: {{ .Values.replicaCount }}  template:    spec:      containers:        - name: api          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
Shell
helm template shop ./chart -f values-prod.yaml     # render only; review the YAMLhelm upgrade --install shop ./chart -f values-prod.yamlhelm history shophelm rollback shop 3                               # go back to revision 3

The values-per-environment pattern is the main reason teams use it: one chart, a values-dev.yaml and a values-prod.yaml. A related tool, Kustomize, patches plain YAML with overlays instead of templates; it is built into kubectl (kubectl apply -k). Pick Helm when you install third-party software or need a reusable, versioned package. Pick Kustomize when you own the YAML and only need small per-environment changes.

Gotchas: Helm only knows what it installed, so edits made by hand with kubectl edit get overwritten by the next upgrade. helm upgrade --atomic rolls back on failure in Helm 3; in Helm 4 the flag is named --rollback-on-failure and --atomic is marked deprecated (I checked the Helm source). Do not hide secrets in values.yaml in Git.

7. Serverless vs containers vs PaaS: how to choose

PaaS (Vercel, Render)Serverless functions (Lambda)Containers on ECS/FargateKubernetes (EKS)
You manageCode onlyCode and function settingsImage, task settings, scaling rulesImage, manifests, cluster add-ons, upgrades
Scale to zeroOftenYesNot by defaultNot by default
Cold startSomeYesNo (pods stay up)No
Long jobs, WebSocketsLimitedHard (time limit)FineFine
Cost at low trafficLowVery lowLow to mediumHighest fixed cost
Cost at steady high loadHighCan be highGoodGood
Team skill neededLowMediumMediumHigh

A cold start is the extra delay when the platform must start a new instance for the first request. See Lesson 26 for how to reduce it.

My rule: start with the least infrastructure that meets the need, and move down the table only when something forces you.

  • A Next.js app with normal traffic and a small team: PaaS. Your time is more expensive than the bill.
  • Spiky or event-driven work (process an upload, send an email, run a nightly job): Lambda, with SQS in front.
  • A steady API, workers, WebSockets, long requests, or a need for a custom runtime: containers on ECS with Fargate.
  • Many services, many teams, a need for one portable platform, custom controllers, or a skilled platform team: Kubernetes. Without that team, Kubernetes is mostly extra work (upgrades every few months, add-ons, networking, security).

ECS vs EKS in one line: ECS is AWS's own simple container scheduler (you write a task definition); EKS is AWS-managed Kubernetes (you write manifests and keep your skills portable). Both can run on Fargate (no nodes to manage) or on EC2 nodes you manage. EKS also charges a fixed hourly fee per cluster (about $73 a month at the list price when I last knew it; check the current price).

8. Core AWS services for a full-stack developer

You do not need the whole catalogue. This is the short list and what each one is for. Hands-on steps for most are in the DevOps series.

ServiceWhat it isUse it forGotcha
EC2Virtual serversFull control, special softwareYou patch and scale it (Lesson 7)
ECS / FargateRuns containers; Fargate hides the serversYour API and workersPay for the task size even when idle
LambdaRuns a function on an event, billed by useGlue code, event handlers, spiky workTime limit per call (up to 15 minutes), cold starts, many connections to a database
S3Object storage (files by key)Uploads, static files, backups, logsKeep buckets private; use pre-signed URLs for access
RDSManaged relational databasePostgres or MySQL with backups and patchingMulti-AZ is for failover, not for reads (Lesson 8)
CloudFrontCDN: caches content near usersAssets and cached pages (Lesson 11)Cache invalidation and cache keys
SQSMessage queueDecouple a slow job from a requestAt-least-once delivery, so handlers must be idempotent
IAMWho can do whatEvery call to AWSToo-wide policies (Lesson 5)
VPCYour private networkPut the database in a private subnet (Lesson 6)NAT gateways cost money

Notes that interviewers like:

  • Pre-signed URL. A pre-signed URL is a temporary link that lets someone upload to or download from one S3 object without AWS keys. The browser sends the file straight to S3, so your server does not carry the bytes.
  • SQS and Lambda. An SQS queue can trigger a Lambda function with a batch of messages. A dead-letter queue (DLQ) is a second queue where messages go after too many failed tries, so one bad message does not block the rest. Return partial batch failures so only the failed messages are retried. Set the queue's visibility timeout longer than the function timeout (AWS advises a multiple of it). The queue concepts are in Queues and Webhooks, Lesson 19.
  • Lambda and databases. Each Lambda instance opens its own database connection, so a traffic spike can open hundreds. Use a connection pooler (RDS Proxy or the database's own pooler) or a database that accepts many connections.
  • S3 is not a file system. It stores whole objects under keys; there are no real folders and no partial edits.

A Lambda handler for SQS with partial batch failures. I compiled this with TypeScript in strict mode against @types/aws-lambda:

JavaScript
import { SQSEvent, SQSBatchResponse } from "aws-lambda";export const handler = async (event: SQSEvent): Promise<SQSBatchResponse> => {  const failures: { itemIdentifier: string }[] = [];  for (const record of event.Records) {    try {      await sendEmail(JSON.parse(record.body));    } catch {      failures.push({ itemIdentifier: record.messageId });    }  }  return { batchItemFailures: failures };};declare function sendEmail(job: unknown): Promise<void>;

(For the function to use this, the SQS event source mapping must have partial batch responses turned on.)

IAM least privilege

The two default rules from Lesson 5: everything is denied unless a policy allows it, and an explicit Deny always wins. Least privilege means the policy names the exact actions, the exact resources, and where possible a condition.

A policy for the API's task role. It can read and write one bucket's objects and send to one queue, nothing else (the JSON parses; the account number is made up):

JavaScript
{  "Version": "2012-10-17",  "Statement": [    {      "Sid": "UploadsReadWrite",      "Effect": "Allow",      "Action": ["s3:GetObject", "s3:PutObject"],      "Resource": "arn:aws:s3:::acme-shop-prod-uploads/*"    },    {      "Sid": "SendEmailJobs",      "Effect": "Allow",      "Action": "sqs:SendMessage",      "Resource": "arn:aws:sqs:eu-west-1:123456789012:shop-prod-emails"    }  ]}

Two kinds of policy matter for every role. The trust policy says who may become the role (for example ecs-tasks.amazonaws.com). The permission policy says what the role may do. In ECS, the task execution role lets ECS itself pull the image and read secrets at start, and the task role is what your code uses at run time. Mixing them up is a common mistake.

"Action": "*" with "Resource": "*" is the one policy every reviewer rejects. When you do not know the needed actions, start narrow, run the app, read the AccessDenied messages (or use IAM Access Analyzer's policy generation from CloudTrail logs), and widen only what is required. Never put long-lived access keys in code or in a container image; give the workload a role.

9. Terraform basics

Terraform is a tool that reads .tf files describing infrastructure and calls cloud APIs to make the real world match. The language is HCL. OpenTofu is a community fork of Terraform with a very similar workflow; the ideas here apply to both. Step-by-step setup is in DevOps, Lesson 15.

Words:

  • A provider is a plugin that talks to one platform (AWS, Cloudflare, Kubernetes).
  • A resource is one thing to create, such as a bucket.
  • State is a file where Terraform records what it created and the real IDs. It is how Terraform knows that aws_s3_bucket.uploads in code is bucket abc in AWS.
  • A module is a folder of Terraform files you call like a function, with inputs (variables) and outputs.
  • Drift is when the real infrastructure no longer matches the code or the state, usually because someone changed it by hand in the console.

The loop

Shell
terraform init      # download providers, set up the backendterraform plan      # show what WOULD change (changes nothing)terraform apply     # make the change (asks to confirm)

plan compares three things: your code, the state, and (after a refresh) the real world. Read the plan like a code review. + is create, ~ is update in place, -/+ is destroy and recreate, which on a database means data loss. Save a plan with -out and apply exactly that file in CI, so what was reviewed is what runs.

I ran a small module with a validation rule. Here is the part that proves the guard works:

HCL
variable "instance_count" {  type    = number  default = 2  validation {    condition     = var.instance_count >= 1 && var.instance_count <= 20    error_message = "instance_count must be between 1 and 20."  }}

Setting instance_count = 50 made terraform plan stop with instance_count must be between 1 and 20 before anything was created.

State: the part that bites

State can contain secrets in plain text (database passwords, for example). Treat it like a secret file. In a team:

  • Keep state remote, for example in an S3 bucket with versioning and encryption, not on one laptop and not in Git.
  • Use locking, so two people (or two pipelines) cannot apply at the same time. The S3 backend supports a lock file with use_lockfile = true in recent Terraform versions (my Terraform 1.16 accepted it); older setups used a DynamoDB table for the lock.
  • Split state by environment and by layer (network, data, app). One giant state is slow and risky.
HCL
terraform {  backend "s3" {    bucket       = "acme-tf-state"    key          = "shop/prod/terraform.tfstate"    region       = "eu-west-1"    encrypt      = true    use_lockfile = true  }}

Drift

Drift is found by terraform plan (it refreshes first). If someone widened a security group by hand, the plan shows the change needed to put it back. terraform plan -refresh-only shows only the difference between state and the real world, so you can decide whether to accept it. You can also bring existing hand-made resources under Terraform with an import block. The fix for drift is cultural as much as technical: remove console write access for most people and let the pipeline apply.

Modules, and two gotchas I tested

A module takes inputs and returns outputs, so dev and prod call the same code with different values. Keep modules small, version them, and do not make them wrap every provider argument.

count versus for_each. With count, resources are tracked by position. I created three items web, api, worker, then removed web from the list. Plan result: with count, two items were updated in place and the last was destroyed. With for_each, tracked by key, only web was destroyed. For AWS resources, that difference can be a recreated database or load balancer. Prefer for_each with stable keys.

Renaming. If you rename a resource in code, Terraform sees "destroy the old, create the new". Add a moved block instead:

HCL
moved {  from = terraform_data.by_key  to   = terraform_data.renamed}

The plan then said each item "has moved" and ended with 0 to add, 0 to change, 0 to destroy.

Also know lifecycle { prevent_destroy = true } (refuse to destroy this, good for databases) and terraform destroy (removes everything in that state, so be careful which directory you are in).

A short AWS example

HCL
resource "aws_s3_bucket" "uploads" {  bucket = "acme-shop-${var.env}-uploads"}resource "aws_s3_bucket_public_access_block" "uploads" {  bucket                  = aws_s3_bucket.uploads.id  block_public_acls       = true  block_public_policy     = true  ignore_public_acls      = true  restrict_public_buckets = true}resource "aws_sqs_queue" "emails_dlq" {  name                      = "shop-${var.env}-emails-dlq"  message_retention_seconds = 1209600}resource "aws_sqs_queue" "emails" {  name                       = "shop-${var.env}-emails"  visibility_timeout_seconds = 60  redrive_policy = jsonencode({    deadLetterTargetArn = aws_sqs_queue.emails_dlq.arn    maxReceiveCount     = 5  })}data "aws_iam_policy_document" "api_task" {  statement {    actions   = ["s3:GetObject", "s3:PutObject"]    resources = ["${aws_s3_bucket.uploads.arn}/*"]  }  statement {    actions   = ["sqs:SendMessage"]    resources = [aws_sqs_queue.emails.arn]  }}

Honest limit: I could only check this file with terraform fmt (syntax and style), because the AWS provider could not be downloaded in my sandbox, so I did not run terraform validate on it. In real work, fmt -check, validate and plan all run in CI on every pull request.

Notice how the IAM policy is built from the resources' own ARNs. Least privilege is easier in code: the policy cannot point at the wrong bucket by typo.

10. Secrets

A secret is any value that gives access: passwords, API keys, tokens, private keys. Rules:

  1. Never in Git, never in an image, never in a NEXT_PUBLIC_* variable (see Lesson 26).
  2. Prefer roles over keys: a task with an IAM role gets short-lived credentials from AWS, and nothing needs to be stored.
  3. When a value must exist, keep it in a secret store and give the app read access to just that value.
PlaceGood forNotes
AWS Secrets ManagerDatabase passwords, API keys, rotationCosts per secret; can rotate automatically
SSM Parameter StoreConfig and cheap secretsSecureString is encrypted; no built-in rotation of the same kind
Kubernetes SecretDelivering a value to a podSee the warning below
External Secrets OperatorSync from Secrets Manager into K8s SecretsKeeps the source of truth outside the cluster
Sealed Secrets / SOPSSecrets that live encrypted in GitOnly the cluster (or key holder) can decrypt

A Kubernetes Secret is not encrypted by default. The values are only base64-encoded, which is a format, not protection. The Kubernetes docs say Secrets are stored unencrypted in etcd by default, and that anyone who can create a pod in a namespace can read the Secrets there. Turn on encryption at rest, restrict access with RBAC, and consider an external secret store.

Other points: secrets in environment variables can leak through crash dumps and debug endpoints, so some teams mount them as files instead. Rotation matters: if a key leaks, you must be able to change it in minutes, so test rotation before you need it. And remember that Terraform state may hold secret values, so lock it down.

11. Cost awareness

Cloud bills punish things you forget. The basics are in DevOps, Lesson 19; here is the interview version.

  • Tag every resource (env, team, service) so the bill can be split. Tags in Terraform make this automatic (default_tags on the provider).
  • Set a budget alarm the first day. The cheapest outage is the one a bill alert catches.
  • Biggest hidden costs: NAT gateway hours and data, data transfer out to the internet, idle load balancers and databases, unattached volumes, and logs kept forever. See Lesson 6 for NAT.
  • Right-size using real CPU and memory graphs, not guesses. Requests that are far above real use waste money in Kubernetes because you pay for the reserved nodes.
  • Shut down what you do not use: dev and staging at night, preview environments after a pull request closes.
  • Commit later, not first: Savings Plans or Reserved Instances only after usage is stable.
  • Pick the model that matches the load. Lambda is cheap for rare work but can cost more than a container under constant high load. Kubernetes has a fixed cost (control plane, add-ons, the people) that only pays off at scale.

A good answer also names the human cost: an engineer's day costs more than most small savings, so do not spend a week to save twenty dollars.

12. Multi-environment strategy

Typical set: dev (fast, cheap, may break), staging (like production, for the last checks), production. Some teams add preview environments per pull request.

What I do:

  • Build once, promote the same image. The image from CI moves from dev to staging to prod. Only the configuration changes. Rebuilding for each environment means you did not test what you ship.
  • One codebase for infrastructure, many inputs. The same Terraform module with dev.tfvars and prod.tfvars, and a separate state per environment. Separate folders or separate pipelines are easier to review than Terraform workspaces, which hide which environment you are in.
  • Separate AWS accounts for prod and non-prod (grouped under AWS Organizations). An account is the strongest wall: a mistake or a leaked key in dev cannot touch prod data. It also makes the bill and the permissions cleaner.
  • Configuration through environment variables or a values file, never if (env === "prod") scattered in code.
  • Keep staging close to prod in shape (same engine version, same migrations), but smaller in size. Do not copy real customer data into staging without masking it.
  • Promotion with approval: automatic to dev, a check to staging, a human click (or a gate on health metrics) to prod.
  • GitOps (an approach where Git is the source of truth for what runs, and a tool inside the cluster such as Argo CD or Flux applies it) gives you an audit trail and a rollback by git revert. It is optional; many teams run helm upgrade from CI and are fine.

A short rule I repeat: same artifact, same code, different configuration, separate state, separate blast radius.

Common mistakes

  • Using :latest image tags, so nobody knows what is running.
  • No readiness probe, so rolling updates send traffic to pods that are not ready.
  • A database check in the liveness probe, causing restart storms.
  • No resource requests, then wondering why pods go Pending or why the HPA does nothing.
  • Memory limit below real use, giving OOMKilled loops.
  • Treating a base64 Secret as safe.
  • Action: * in IAM, and long-lived access keys in code.
  • Using count for things that have an identity.
  • Editing infrastructure in the console and leaving drift.
  • Choosing Kubernetes for a team of three because it looks good on a CV.

Interview questions

(Must know) What is the difference between a container and a virtual machine?

A VM runs a full operating system with its own kernel on virtualized hardware. A container is a normal process that the host kernel isolates with namespaces (what it can see) and cgroups (how much it can use). So containers share the host kernel. That makes them small and fast to start, usually a second or less, and lets me pack many on one machine. The trade-off is weaker isolation: a kernel flaw can affect all containers on the host, and I cannot run a different OS kernel in a container. In practice I run containers on top of VMs, either EC2 nodes or Fargate, which hides the VMs.

(Must know) Explain Pod, Deployment, ReplicaSet and Service. How do they relate?

A pod is one or more containers sharing a network address; it is disposable. A Deployment says "keep N copies of this pod template running", and it handles updates. For each version, the Deployment creates a ReplicaSet, which does the counting and replaces pods that die. A Service gives a stable name and IP in front of those pods and finds them by label selector, because pod IPs change. So: Deployment owns ReplicaSets, ReplicaSets own pods, and a Service points at pods by label. I would only create a bare pod for a quick debug.

(Must know) What is the difference between a liveness, a readiness and a startup probe?

Readiness asks "can you take traffic now?" and when it fails the pod is removed from the Service, with no restart. Liveness asks "are you stuck forever?" and when it fails the container is restarted. Startup asks "have you finished booting?" and holds off the other two until it passes, which helps slow starters like a big Node app. The key point: keep liveness simple (the process answers), and put dependency checks in readiness if anywhere.

(Trap) Your liveness probe calls the database. What can go wrong?

If the database slows down or has a short outage, every pod fails liveness at the same time. Kubernetes restarts all of them. During restarts nothing serves traffic, and when they come back they all open new database connections at once, which can keep the database down. A database problem becomes a full outage with extra load. The fix: liveness only checks the process itself. For a dependency, I use readiness (so pods stop getting traffic but are not killed), or a degraded response, and I add retries and a circuit breaker in the app.

(Must know) What are requests and limits, and what happens when a container goes over each?

A request is what the scheduler reserves for the container; it picks a node with that much free, and the HPA uses it as the base for percentages. A limit is the maximum. Over the CPU limit, the container is throttled: slower, but alive. Over the memory limit, it is killed with OOMKilled (exit code 137) and restarted. That is why I treat memory limits carefully. My default is memory request equal to limit, a CPU request, and often no CPU limit to avoid throttling spikes, though I would follow a team policy that says otherwise.

(Senior signal) A pod is in CrashLoopBackOff. Walk me through your debugging.

I start with kubectl get pods for the state, then kubectl describe pod for Events and the Last State. That tells me the exit code and the reason: OOMKilled means memory; a probe failure shows in Events; an exit code 1 means the app itself crashed. Then kubectl logs <pod> --previous, because the current container just started and the old one has the error. Common causes are a missing environment variable or Secret, a wrong database URL, a bad command, or a liveness probe that is too strict for a slow start (fix with a startup probe). If the image has no shell, I use kubectl debug with a debug container. I fix the cause and check with kubectl rollout status.

(Must know) How does a rolling update work, and what do maxSurge and maxUnavailable do?

The Deployment starts a new ReplicaSet and scales it up while scaling the old one down. maxSurge is how many extra pods may exist above the desired count; maxUnavailable is how many may be missing. Both default to 25%. With maxSurge: 1 and maxUnavailable: 0, it adds one new pod, waits until it is ready, then removes one old pod. That keeps full capacity but is slower and needs spare room on the nodes. It only works safely when the readiness probe is accurate. If the new pods never become ready, the old ones stay up, and after the progress deadline (600 seconds by default) the rollout is marked failed.

(Senior signal) A bad version is live. How do you roll back, and what does a rollback not fix?

First I stop the damage: kubectl rollout undo deploy/shop-api, which returns to the previous ReplicaSet in seconds, or helm rollback if Helm owns it, or git revert in a GitOps flow. Then I verify with rollout status and the dashboards. A rollback restores the code and config of the workload. It does not restore data. If the release ran a destructive migration, wrote bad data, or sent emails, I need more steps. That is why I use expand-then-contract migrations and feature flags: the flag turns a feature off without any deploy. I also keep enough revision history (revisionHistoryLimit) so there is something to go back to.

(Must know) What does Helm give you over plain YAML, and when would you use Kustomize instead?

Helm gives templates with values, so one chart serves many environments, plus a release history with helm rollback and a way to install third-party software (Ingress controllers, monitoring) in one command. The cost is a template language that is harder to read than plain YAML. Kustomize patches plain YAML with overlays and is built into kubectl. I use Helm for third-party packages and for charts that many teams share, and Kustomize when I own the YAML and need small changes per environment. A Helm detail I check: helm template shows the final YAML for review, and in Helm 4 the old --atomic flag is deprecated in favor of --rollback-on-failure.

(Trap) Is a Kubernetes Secret safe because it is base64 encoded?

No. Base64 is just a text encoding; anyone can decode it in one command. By default Secrets are stored unencrypted in etcd, and anyone who can read Secrets in the namespace, or create a pod there, can get them. To make them safer I turn on encryption at rest, limit access with RBAC, avoid committing Secret YAML to Git, and often keep the real value in AWS Secrets Manager and sync it with the External Secrets Operator, or use Sealed Secrets or SOPS for encrypted-in-Git. Best of all, I avoid long-lived secrets by using IAM roles for AWS access.

(Senior signal) How does the HPA decide the number of replicas, and why might it do nothing?

It checks the metric on an interval (15 seconds by default) and uses desired = ceil(current replicas x current value / target value). With 3 pods at 90% CPU and a target of 70%, that is ceil(3 x 90/70) = 4. A small tolerance (10% by default) avoids tiny changes, and scale-down waits for a stabilization window (5 minutes by default) to stop flapping. It does nothing when: Metrics Server is not installed, the pods have no CPU request (so the percentage has no base), the bottleneck is not CPU (for example the database or a queue), or the app is already at maxReplicas. For queue workers I would scale on queue length with KEDA or a custom metric, not CPU. And scaling pods needs node capacity too, so the cluster must add nodes.

(Senior signal) Choose a platform: a Next.js marketing site, an event-driven image processor, and a steady API with WebSockets for 40 engineers' product.

The marketing site goes on a PaaS like Vercel: no operations, global CDN, instant rollbacks. The image processor fits S3 events with SQS and Lambda: it scales to zero, costs little when idle, and each job is short; I set a dead-letter queue and partial batch failures. The API with WebSockets needs long-lived connections and steady load, so containers: ECS with Fargate if one team owns it and the needs are simple. If many teams run dozens of services and need shared tooling, I would consider EKS, but only with a platform team to run it. I would say that Kubernetes is a cost I must justify, not a default.

(Must know) What is IAM least privilege, and how would you apply it to an API that uploads files to S3 and sends to a queue?

Least privilege means each identity gets exactly the actions on exactly the resources it needs. For the API I create a role for the ECS task with a policy that allows s3:GetObject and s3:PutObject on the uploads bucket's objects only, and sqs:SendMessage on one queue ARN. No * actions, no * resources. The trust policy allows only the ECS tasks service to assume the role. I keep the task role (what my code can do) separate from the execution role (what ECS needs to pull the image and read secrets). If I am unsure what is needed, I start narrow, read the AccessDenied errors in tests, and widen one action at a time. For deploys from GitHub Actions I use OIDC with a narrow trust rule, not stored keys (Lesson 26).

(Senior signal) Lambda is calling Postgres and the database is running out of connections during a spike. Why, and what do you do?

Each concurrent Lambda instance has its own process and its own connection, so 500 concurrent invocations can try to open 500 connections. Postgres handles that badly. Options: put a pooler in front (RDS Proxy or PgBouncer), keep a small pool size of 1 per instance, create the client outside the handler so warm instances reuse it, set reserved concurrency on the function to cap the load, or move to a database with an HTTP or pooled endpoint. If the work is heavy and steady, a long-running container with one shared pool is the simpler design. This is a good example of "serverless is not free of design cost".

(Must know) What is Terraform state, and what do you do about it on a team?

State is the file where Terraform stores which real resources match which resources in code, with their IDs and attributes. Without it, Terraform cannot know what it already created. It can contain secrets in plain text. On a team I keep it remote (S3 with versioning and encryption), turn on locking so two applies cannot run together (a lock file in the bucket with newer Terraform, or a DynamoDB table in older setups), restrict who can read it, and split it by environment and layer so a mistake has a small blast radius. I never edit it by hand; I use terraform state mv, moved blocks and import blocks.

(Trap) A teammate removed the first item from a list used with count, and plan wants to destroy and recreate things. Why?

With count, Terraform identifies each resource by its index. If the list loses its first item, every later item shifts down one place, so item at index 1 is now seen as index 0, and so on. Terraform updates or replaces those and destroys the last index. I tested this: removing web from [web, api, worker] gave two in-place updates and one destroy with count, but with for_each only the web entry was destroyed. For anything with an identity (buckets, queues, subnets) I use for_each with stable keys. If I must rename or move a resource, I use a moved block, which gave 0 to destroy in my test.

(Senior signal) What is drift, how do you detect it, and how do you prevent it?

Drift is when the real infrastructure differs from the code and state, usually from a manual console change or an emergency fix. terraform plan detects it because it refreshes first, and plan -refresh-only shows only that gap. A scheduled plan in CI that alerts on a non-empty result catches drift early. Then I choose: change the code to match the new reality (and apply or import), or apply the code and overwrite the manual change. Prevention is mostly process: remove write access in production consoles for most people, make the pipeline the only path to apply, and keep a break-glass role that is logged. Emergency console fixes are fine if they are turned into code the next day.

(Senior signal) Design a multi-environment setup for a small team on AWS.

Three AWS accounts under Organizations: dev, staging, prod, with prod having the tightest access and a separate bill. One Terraform repo with shared modules; each environment is its own directory (or pipeline) with its own tfvars and its own remote state. CI builds the Docker image once, tags it with the commit, pushes it to a registry, and deploys that same image to dev automatically, then to staging, then to prod after approval. Config comes from environment variables, with secrets in Secrets Manager read by the task role. Staging uses the same database engine and migrations as prod but a smaller size, no real customer data, and shuts down at night. Budgets and tags per environment. The idea is the same artifact in every environment, only configuration differs, and a mistake in one account cannot reach another.

(Trap) "Our AWS bill doubled and nobody changed the code." Where do you look?

I open Cost Explorer grouped by service, then by usage type and by tag, and compare to last month. The usual suspects: NAT gateway data (a job suddenly calling the internet through it), data transfer out, a forgotten large instance or database, a load balancer left running, CloudWatch Logs growing because someone turned on debug logging, S3 request or storage growth with no lifecycle rule, and an autoscaler that scaled up and never came back. Untagged spending is itself a finding. I fix the cause, add a lifecycle or retention rule, and set a budget alert and anomaly detection so we hear about it in days, not at month end. If the cause is real growth, then I talk about right-sizing and commitments.

(Senior signal) Your team says "let's move to Kubernetes". What do you ask before agreeing?

I ask what problem we have that the current setup cannot solve. Do we have many services and teams, a need for portability across clouds, special scheduling, or a platform team to run it? Kubernetes brings upgrades several times a year, add-ons (ingress, certificates, metrics, secrets), networking and security policy, and a steep learning curve for on-call. A managed option like ECS on Fargate or a PaaS may give 90% of the benefit for a fraction of the work. If the answer is still yes, I would start with managed Kubernetes (EKS), a small pilot service, infrastructure in Terraform, a clear owner, and a plan for upgrades and on-call training. I would also write down what success looks like, so we can check it after six months.

Key takeaways

  • A container is an isolated process sharing the host kernel; a VM is a whole machine. Use containers for packing and fast deploys, and know the isolation trade-off.
  • Deployment keeps pods alive and updates them, Service gives a stable address by labels, Ingress (or the newer Gateway API) routes HTTP, HPA changes replica count.
  • Readiness removes a pod from traffic, liveness restarts it, startup protects slow boots. Never check the database in liveness.
  • Set requests always. CPU over the limit is throttled; memory over the limit is OOMKilled.
  • Rolling updates depend on accurate readiness probes. maxSurge and maxUnavailable default to 25%. rollout undo restores code, not data.
  • Helm templates and versions releases; Kustomize patches plain YAML.
  • Choose the simplest platform first: PaaS, Lambda, ECS/Fargate, and Kubernetes last.
  • Least privilege means exact actions, exact resources, roles instead of keys.
  • Terraform: remote state with locking, read every plan, prefer for_each to count, use moved for renames, watch for drift.
  • A Kubernetes Secret is only base64 by default. Use encryption at rest, RBAC or an external secret store.
  • Tag everything, set budgets, and watch NAT, data transfer and forgotten resources.
  • Environments: build once, promote the same image, separate state and separate accounts.

What's next

To build these pieces by hand, follow the DevOps series: IAM, VPC, Docker, Auto Scaling, Terraform and Cost. For the release side, return to Deployment, CI/CD and Monitoring. Then practise Q6 and Q18 out loud, using a real project you have worked on.

Published by Jainik · Last updated · How these lessons are made

Keep learning

Related lessons

Part of the study guide

Related series