Docker & Kubernetes

Services, Ingress and Networking in Kubernetes

How pods find each other, ClusterIP, NodePort and LoadBalancer, DNS names, Ingress, and what a service mesh is for.

Lesson 0514 min read#kubernetes#services#ingress#networking

On this page

  1. Words used in this lesson
  2. The problem: pods have short lives and moving addresses
  3. Labels and selectors: how Kubernetes "points" at pods
  4. Your first Service: ClusterIP
  5. How the Service finds its pods: EndpointSlices
  6. Service types
  7. DNS names: how a pod finds a Service
  8. What really happens: kube-proxy and the virtual IP
  9. Ingress: HTTP routes from outside
  10. Reaching a Service from your laptop
  11. NetworkPolicy: who may talk to whom
  12. What a service mesh is for (and why you probably do not need one yet)
  13. A real mini lab you can do without a cluster
  14. Commands to know (reviewed, not run on a cluster here)
  15. Try it
  16. Interview questions
  17. Key takeaways
  18. What's next

In one line: pods come and go and their IP addresses change, so Kubernetes gives you a Service, a stable name and address that always points at whichever pods are healthy right now, and an Ingress (or its newer sibling, the Gateway API) to let HTTP traffic from the outside world reach those Services.

In the last lesson you made a Deployment. It keeps three copies of your app running. That is great, but there is a problem you have probably already seen: if the pods can be killed and replaced at any time, how does anything find them? You cannot write http://10.244.1.7:3000 in your code, because that pod may be gone in ten minutes and its replacement will have a different address.

This lesson solves that problem step by step. First we meet the Service, which is the answer inside the cluster. Then we meet the Ingress, which is the answer from outside the cluster. Along the way we look at DNS names, the four Service types, headless Services, network policies, and what a "service mesh" is for.

This is lesson 5 of 9. It builds on Kubernetes Core Ideas: Cluster, Pods and Deployments, where we made the notes-api Deployment. The next lesson, Configuration, Secrets and Storage, shows how to give those pods settings and disk space. If you want the Docker-only version of networking first, Volumes, Networks and Running Containers Safely covers ports and user-defined networks.

What I ran / checked. Tools on my machine: Docker Engine 29.8.2 (a real daemon, so the Docker examples below are real runs), kubectl v1.37.1 (client only), kubeconform v0.8.0 (checks YAML against the official Kubernetes JSON schemas, here for Kubernetes 1.34.0), yamllint 1.38.0 and kind v0.33.0. I could not run a real Kubernetes cluster. I pulled the kind node image (kindest/node:v1.34.0) and tried to create a cluster. kind's control plane never became healthy, because the nested container runtime inside the sandbox could not start the control plane pods (the kubelet reported a runc create failed ... can't get final child's PID from pipe: EOF error). So every Kubernetes manifest here is labelled "reviewed, not run on a cluster here". What I did do for real: every manifest passed kubeconform -strict against the 1.34.0 schemas and yamllint, I made kubectl create ... --dry-run=client -o yaml print the YAML it would send, and I deliberately broke manifests to see the real error text. Anything that only a cluster can show (events, pod IPs, kubectl get endpointslices output) I describe in words, and I do not paste fake terminal output. Sources I read (own words below): the Kubernetes website docs on Service, Ingress, Ingress controllers, Gateway API, DNS for Services and Pods, and Network Policies (raw markdown from the kubernetes/website repository on GitHub), plus the official JSON schemas used by kubeconform. Could not reach: the rendered kubernetes.io site, registry.k8s.io and Docker Hub (rate limited), so images came from mirror.gcr.io. I did not read any service mesh project's own documentation, so the mesh section is short and general. I verified no GitHub star counts.

Words used in this lesson

  • Pod IP is the address a single pod gets inside the cluster. It changes when the pod is replaced.
  • Label is a small key and value tag on any Kubernetes object, such as app: notes-api.
  • Selector is a rule that picks objects by their labels, such as "every pod with app: notes-api".
  • Service is a stable name and virtual address in front of a changing set of pods.
  • ClusterIP is the default Service type. The address works only inside the cluster.
  • NodePort is a Service type that also opens one port on every node.
  • LoadBalancer is a Service type that asks the cloud for an external load balancer.
  • Headless Service is a Service with no virtual address. DNS returns the pod addresses directly.
  • EndpointSlice is the list of pod addresses (and whether each is ready) behind a Service.
  • kube-proxy is the node component that makes a Service address actually work.
  • Cluster DNS is the built-in DNS server that turns Service names into addresses.
  • Ingress is an object that describes HTTP and HTTPS routes from outside the cluster to Services.
  • Ingress controller is the program that reads Ingress objects and does the real routing. Without one, an Ingress does nothing.
  • Gateway API is the newer, richer set of objects for routing traffic into a cluster (GatewayClass, Gateway, HTTPRoute).
  • NetworkPolicy is an object that says which pods may talk to which pods.
  • Service mesh is extra infrastructure that adds things like encryption and retries between services.

The problem: pods have short lives and moving addresses

Every pod in a cluster gets its own IP address. Any pod can talk to any other pod using that address, without NAT. That is part of the Kubernetes networking model. So in theory you could just ask for the IP of your database pod and connect.

In practice this breaks at once, for three reasons:

  1. Pods are replaced, not repaired. When a pod dies, the Deployment creates a new pod. The new pod has a new IP.
  2. Scaling changes the set. Today you have three API pods. Tomorrow you have ten. A client should not need to know.
  3. Rolling updates replace every pod. During a rollout all the old addresses disappear.

If you used Docker Compose in lesson 3, you already saw the fix for the same problem on one machine: you connect to db, not to an IP, and Docker's DNS finds it. Kubernetes does the same, with more rules. The object that does it is the Service.

Labels and selectors: how Kubernetes "points" at pods

Before the Service, one idea you must be comfortable with: labels and selectors. Kubernetes almost never says "this exact pod". It says "all pods that look like this".

A label is a key and value you put on an object. In our Deployment from lesson 4, the pod template has:

YAML
template:  metadata:    labels:      app: notes-api

Every pod that Deployment makes carries app: notes-api. A selector is a question about labels. app: notes-api means "give me everything with that label and that value". The Deployment itself uses a selector (spec.selector.matchLabels) to know which pods belong to it. A Service uses exactly the same trick to know which pods to send traffic to.

Here is the Deployment again, so this lesson stands alone. It is the same shape as lesson 4, with a named port.

YAML
# deployment.yamlapiVersion: apps/v1kind: Deploymentmetadata:  name: notes-apispec:  replicas: 3  selector:    matchLabels:      app: notes-api  template:    metadata:      labels:        app: notes-api    spec:      containers:        - name: api          image: notes-api:1.0          ports:            - name: http              containerPort: 3000

notes-api:1.0 is a placeholder for the image you built in lesson 1 and 2 (it would have to be in a registry your cluster can reach). Notice the port has a name, http. That small detail will pay off in a minute.

Status: reviewed, not run on a cluster here. This file passed kubeconform -strict -kubernetes-version 1.34.0 and yamllint. I could not apply it to a real cluster.

Your first Service: ClusterIP

This is the smallest useful Service.

YAML
# svc-clusterip.yamlapiVersion: v1kind: Servicemetadata:  name: notes-apispec:  selector:    app: notes-api  ports:    - name: http      port: 80      targetPort: http

Read it line by line:

  • metadata.name: notes-api is the name other pods will use. It becomes a DNS name (more on that soon).
  • spec.selector says: send traffic to pods labelled app: notes-api.
  • port: 80 is the port the Service listens on. Clients connect to notes-api:80.
  • targetPort: http is the port on the pod where traffic goes. Because we gave the container port the name http, we can use the name instead of the number 3000.
  • No type means ClusterIP, the default.

Status: reviewed, not run on a cluster here. Validated with kubeconform -strict and yamllint. kubectl apply was not possible.

The reason to use a named targetPort is that you can change the container's number later (say from 3000 to 8080) in one place, the Deployment, and the Service keeps working. With a number you must change two files and hope you did not forget one. The Kubernetes docs list this as a benefit of named target ports.

The four port numbers, in one table

The most common beginner confusion is that "port" appears in four places. Here is the whole map:

FieldWhere it livesWhat it means
containerPortIn the pod specThe port your app listens on inside the container. Mostly documentation, it does not open anything.
targetPortIn the ServiceWhich pod port the Service sends traffic to. Number or name.
portIn the ServiceThe port the Service itself listens on. What clients use.
nodePortIn a NodePort ServiceA port opened on every node (default range 30000 to 32767).

If targetPort is left out, Kubernetes uses the same number as port. That is a classic trap: you set port: 80, forget targetPort, and the Service sends traffic to port 80 of a pod that listens on 3000. The connection is refused and nothing in the YAML looks wrong.

What you get from a ClusterIP

When the Service is created, Kubernetes gives it a virtual IP (the cluster IP) that does not change for the life of the Service. Cluster DNS also gets a name for it. A pod that connects to http://notes-api (from the same namespace) reaches one of the three API pods.

Two important things about this address:

  • It is virtual. No network card has this IP. Nothing "listens" on it. The node's networking rules (set up by kube-proxy, see below) rewrite traffic aimed at it so it lands on a real pod.
  • It only works inside the cluster. That is what the name means. From your laptop you cannot reach a ClusterIP. We will fix that in the Ingress and port-forward parts.

How the Service finds its pods: EndpointSlices

You wrote a selector. Who turns that into a list of addresses? A control loop in the control plane. It watches pods and Services, and for each Service it keeps an EndpointSlice object up to date. An EndpointSlice is a list of pod IPs and ports behind a Service, plus a flag for each address that says whether the pod is ready.

Some facts straight from the docs:

  • EndpointSlices are the current API. The older Endpoints object is deprecated (since v1.33). One reason: it cuts off the list when it gets too long for one object.
  • By default, a new EndpointSlice is started once the existing ones each hold at least 100 endpoints. So big Services are split into slices.
  • Only ready pods receive traffic. This is why readiness probes matter (lesson 7). A pod that is running but not ready is in the list, but marked not ready, and clients are not sent to it.

On a real cluster you would look at this with kubectl get endpointslices or kubectl describe service notes-api. I could not run those here, so I will not show output. What to look for is this: if the list of endpoints is empty, no pod matched the selector (or none is ready), and every request to the Service will fail. This is the single most useful check when "the Service doesn't work", and we will use it again in lesson 8.

Here is a Service with a typo in the selector. It is perfectly valid YAML and passes schema checks, which is exactly why it is dangerous:

YAML
# svc-mismatch.yamlapiVersion: v1kind: Servicemetadata:  name: notes-apispec:  selector:    app: notes-apii        # extra "i": matches no pod  ports:    - port: 80      targetPort: 3000
TEXT
$ kubeconform -strict -summary -kubernetes-version 1.34.0 svc-mismatch.yamlSummary: 1 resource found in 1 file - Valid: 1, Invalid: 0, Errors: 0, Skipped: 0

(That output is real.) The schema checker cannot know that no pod has the label app: notes-apii. Only the cluster knows. The result in a real cluster would be a Service with an empty endpoint list. The lesson: a green YAML check does not mean a working app.

Service types

The type field decides how far the Service is reachable. There are four types you should know.

TypeReachable fromHowTypical use
ClusterIP (default)Inside the cluster onlyVirtual IP and DNS nameService-to-service calls, databases
NodePortOutside, via any node's IPAlso opens one port on every nodeQuick tests, or as a building block for a load balancer
LoadBalancerOutside, via an external addressAsks the cloud provider to create a load balancerExposing one service directly in the cloud
ExternalNameInside the clusterDNS alias (a CNAME) to an outside nameGiving a friendly in-cluster name to an outside service

Each builds on the one before it. The docs describe NodePort as "ClusterIP plus a port on every node", and LoadBalancer as "NodePort plus an external load balancer" (on most cloud setups).

NodePort

YAML
# svc-nodeport.yamlapiVersion: v1kind: Servicemetadata:  name: notes-api-nodeportspec:  type: NodePort  selector:    app: notes-api  ports:    - port: 80      targetPort: http      nodePort: 30080

Now http://<any-node-ip>:30080 reaches the API. If you leave nodePort out, Kubernetes picks a free one from the range 30000 to 32767 (the default of the --service-node-port-range flag).

Status: reviewed, not run on a cluster here. Passed kubeconform -strict and yamllint.

Why not use NodePort for production? Because clients must know a node's IP, nodes come and go, and you are exposing a high, odd port on every machine. It is fine for a lab or for a load balancer to sit in front of.

LoadBalancer

YAML
# svc-lb.yamlapiVersion: v1kind: Servicemetadata:  name: notes-api-publicspec:  type: LoadBalancer  selector:    app: notes-api  ports:    - port: 80      targetPort: http

On a cloud provider (or with a controller that supports it) this creates a real external load balancer and writes its address into the Service status. On a laptop cluster with no cloud, the external address often stays <pending> forever, because nothing is there to create it. The docs say the actual load balancing is done by the cloud provider's own system. You pay for each one, so one LoadBalancer per service gets expensive fast. That cost is the main reason Ingress exists.

Status: reviewed, not run on a cluster here. Passed kubeconform -strict and yamllint.

ExternalName

An ExternalName Service has no selector and no pods. It makes cluster DNS return a CNAME to a name you give it, for example db.example.com. It is a naming trick, not a proxy: it does no port changes and no load balancing, and with HTTPS the host name the client sees might not match the certificate. Use it rarely.

Headless Services: no virtual IP at all

Sometimes you do not want one virtual address. You want to know every pod, for example for a database cluster where each member must talk to specific members. Setting clusterIP: None makes a headless Service:

YAML
# svc-headless.yamlapiVersion: v1kind: Servicemetadata:  name: notes-api-podsspec:  clusterIP: None  selector:    app: notes-api  ports:    - port: 3000

The docs say: for a headless Service, no cluster IP is allocated, kube-proxy does not handle it, and there is no load balancing done by the platform. Instead, cluster DNS answers a lookup of the Service name with the addresses of all ready pods. The client chooses. StatefulSets (lesson 6) depend on this.

Status: reviewed, not run on a cluster here. Passed kubeconform -strict and yamllint.

Note that clusterIP: None is a special string. The docs warn that it is not the same as leaving the field out.

I cannot run cluster DNS here, but Docker has the same idea in a smaller form, and I can show it for real. If several containers on a user-defined network share one network alias, Docker's built-in DNS returns all of their addresses:

Shell
docker network create demo-netfor n in 1 2 3; do  docker run -d --rm --name web$n --network demo-net --network-alias web \    mirror.gcr.io/library/busybox:1.36 sh -c "echo web$n > /i; httpd -f -p 8080 -h /"donedocker run --rm --network demo-net mirror.gcr.io/library/busybox:1.36 sh -c \  'nslookup web 2>&1 | tail -8; for i in 1 2 3 4 5 6; do wget -qO- http://web:8080/i; done'
TEXT
Non-authoritative answer:Name:	webAddress: 172.19.0.2Name:	webAddress: 172.19.0.4Name:	webAddress: 172.19.0.3web3web1web1web1web1web3

One name, three addresses, and the client ended up at different containers (the order and mix change from run to run). This is client-side choice over DNS, the same idea as a headless Service. It also shows the weakness: the client picked web1 four times out of six. DNS does not balance load evenly, and clients may cache the answer. A normal ClusterIP Service does the balancing on the node instead.

DNS names: how a pod finds a Service

Cluster DNS gives every Service a name. The full form is:

TEXT
<service>.<namespace>.svc.cluster.local

So our Service in the default namespace is notes-api.default.svc.cluster.local. You do not have to type all that. Each pod gets a /etc/resolv.conf with a search list. The docs show it like this (for a pod in some namespace):

TEXT
search <namespace>.svc.cluster.local svc.cluster.local cluster.localoptions ndots:5

Because of the search list:

  • From a pod in the same namespace, notes-api is enough.
  • From a pod in another namespace, use notes-api.default (the namespace is part of the name).
  • The long form always works.

The ndots:5 option means: if a name has fewer than five dots, try the search list first. This is helpful, but it has a side effect: a lookup for an outside name like api.stripe.com (two dots) may first try several wrong names such as api.stripe.com.default.svc.cluster.local. That is extra DNS traffic. If you see lots of DNS lookups for outside names, adding a trailing dot (api.stripe.com.) makes the name absolute. I name this as a known detail of the setup; I did not measure its cost.

Named ports also get SRV records, which return both the address and the port number. You will rarely use them by hand.

Do not use IP addresses of Services or pods in config files. Use the DNS name. The Service IP is stable for the Service's life, but if you delete and recreate the Service, it may change. The name never does.

The older way: environment variables

Kubernetes also injects environment variables like NOTES_API_SERVICE_HOST into pods. The docs mention them as the second discovery method after DNS. They have a catch: they are set when the pod starts, so a Service created after the pod is invisible to it. DNS has no such problem. Prefer DNS.

What really happens: kube-proxy and the virtual IP

A Service IP that no machine owns sounds like magic, so let us take the magic out. Each node runs a component called kube-proxy. It watches Services and EndpointSlices and programs the node's own packet rules (iptables, IPVS or nftables, depending on the setup) so that packets sent to the Service IP and port are rewritten to the IP and port of one chosen ready pod. The choice happens in the kernel, on the node where the client pod runs.

Two consequences are worth remembering:

  • There is no central proxy server for ClusterIP traffic. Each node does the rewriting for its own pods. That is why it scales well and why you cannot ping a Service IP: ping uses a protocol the rules do not rewrite.
  • Headless Services skip all this, as the docs say. DNS gives the real pod IPs.

The docs also describe knobs that change which backends are chosen: internalTrafficPolicy and externalTrafficPolicy (for example, only use pods on the same node), trafficDistribution (prefer pods in the same zone), and session affinity by client IP. They are advanced. The key point for now: by default the spread is simple and not smart. It does not look at how busy each pod is.

A Service is a connection-level load balancer, not an HTTP one. It picks a pod when a connection opens. With long-lived connections (gRPC, HTTP/2, WebSockets), one client can keep talking to one pod for hours, so a new pod you scaled up may get no traffic from existing clients. For those protocols you may need a smarter proxy, a mesh, or client-side balancing.

Ingress: HTTP routes from outside

Now the outside world. One LoadBalancer per service is expensive and clumsy: you get one IP per service, no host names, no paths, no TLS handling. Ingress solves this for HTTP and HTTPS. It is a set of rules: "requests for host X and path Y go to Service Z". One entry point, many Services behind it.

Here is the key idea, which trips up many people: an Ingress is only a set of rules, written as a Kubernetes object. It does nothing by itself. You must run an Ingress controller, a program that reads Ingress objects and configures a real proxy. The docs say it plainly: only creating an Ingress resource has no effect. Common controllers are proxies like NGINX, HAProxy, Traefik and cloud load balancers. Each one behaves slightly differently, and the docs warn you to read the controller's own documentation.

Here is a complete Ingress:

YAML
# ingress.yamlapiVersion: networking.k8s.io/v1kind: Ingressmetadata:  name: notesspec:  ingressClassName: nginx  tls:    - hosts:        - notes.example.com      secretName: notes-tls  rules:    - host: notes.example.com      http:        paths:          - path: /api            pathType: Prefix            backend:              service:                name: notes-api                port:                  number: 80          - path: /            pathType: Prefix            backend:              service:                name: notes-web                port:                  number: 80

Status: reviewed, not run on a cluster here. Passed kubeconform -strict and yamllint. It assumes a controller whose class is named nginx, a Service called notes-web, and a TLS Secret named notes-tls that I did not create.

How to read it:

  • ingressClassName: nginx says which controller should handle this Ingress. If a cluster has several controllers, this chooses. If you leave it out, the cluster needs a default IngressClass, or no controller may pick it up. The docs recommend always defining a default class.
  • rules[].host is the host name. Both host and path must match before traffic is sent on.
  • Each path has a backend that names a Service and port, not a pod. Ingress sits on top of Services.
  • tls says: for these hosts, terminate HTTPS using the certificate in the Secret notes-tls. The controller does the TLS work, so your pods can speak plain HTTP inside the cluster.

You can also ask kubectl to write the skeleton for you. This is real output from kubectl create ingress with --dry-run=client:

TEXT
$ kubectl create ingress web --rule="notes.example.com/api*=notes-api:80" --class=nginx --dry-run=client -o yamlapiVersion: networking.k8s.io/v1kind: Ingressmetadata:  name: webspec:  ingressClassName: nginx  rules:  - host: notes.example.com    http:      paths:      - backend:          service:            name: notes-api            port:              number: 80        path: /api        pathType: Prefixstatus:  loadBalancer: {}

Notice that the trailing * in the rule turned into pathType: Prefix. That is a handy way to avoid hand-typing nested YAML.

pathType: the rule people get wrong

pathType is required in networking.k8s.io/v1. If you forget it, the object is rejected. Here is the real schema error from my test, with a missing pathType and a port number written as a string ("80"):

TEXT
$ kubeconform -strict -summary -kubernetes-version 1.34.0 ing-nopath.yamling-nopath.yaml - Ingress bad is invalid: ... at '/spec/rules/0/http/paths/0': missing property 'pathType'  - at '/spec/rules/0/http/paths/0/backend/service/port/number': got string, want null or integerSummary: 1 resource found in 1 file - Valid: 0, Invalid: 1, Errors: 0, Skipped: 0

(I shortened the line. The two errors are real and both were found in one run.)

There are three path types:

pathTypeMeaning
ExactMatches the URL path exactly, and is case sensitive.
PrefixMatches by path prefix, split on /, element by element.
ImplementationSpecificUp to the controller. It may treat it like Prefix, Exact, or something else.

Prefix works on whole path elements, not on characters. A few examples taken from the docs table: a Prefix of /foo matches /foo and /foo/. A Prefix of /aaa/bb does not match /aaa/bbb, because bb and bbb are different elements. A Prefix of / matches every path. When more than one rule matches, the longest match wins, and if equal, Exact wins over Prefix.

Ingress annotations are not portable. Features like rewrites, rate limits, timeouts and body size limits are set with annotations that belong to one controller. The same YAML on a different controller silently ignores them. This is one of the reasons the Kubernetes project now points people toward Gateway API.

Ingress is frozen: Gateway API is the future

This is a recent change worth knowing for interviews. The current Kubernetes docs put a note at the top of the Ingress page: the project recommends using Gateway instead of Ingress, and the Ingress API has been frozen. That means:

  • Ingress is generally available and stable, with the usual stability promises.
  • The project has no plans to remove it.
  • But it will get no new features or changes.

So Ingress is not going away, and you will see it in almost every existing cluster. New designs, though, should look at the Gateway API.

Gateway API is a family of resource kinds, installed as an add-on (custom resources), not built into the core API. The docs list four stable kinds:

KindWhat it is
GatewayClassA type of gateway, managed by a controller. Like IngressClass.
GatewayAn actual instance of traffic-handling infrastructure, such as a cloud load balancer.
HTTPRouteHTTP routing rules from a Gateway to backends (usually Services).
GRPCRouteThe same for gRPC.

The design goals in the docs: role-oriented (infrastructure team owns the Gateway, app team owns the routes), portable (the same objects across implementations), expressive (header matching and traffic weights without custom annotations) and extensible. Here is the route that matches our Ingress /api rule:

YAML
# gateway.yamlapiVersion: gateway.networking.k8s.io/v1kind: HTTPRoutemetadata:  name: notesspec:  parentRefs:    - name: shared-gateway  hostnames:    - notes.example.com  rules:    - matches:        - path:            type: PathPrefix            value: /api      backendRefs:        - name: notes-api          port: 80

Status: reviewed, not run on a cluster here. This is not a core type, so I checked it with kubeconform using the community CRD schema catalogue (datreeio/CRDs-catalog) for HTTPRoute. It passed. It assumes a Gateway called shared-gateway already exists and that the Gateway API CRDs and a controller are installed. Neither exists in my sandbox.

The point of the split is ownership. A platform team creates the Gateway once (ports, TLS, which namespaces may attach). Each app team writes only its own HTTPRoute, in its own namespace. With Ingress, one object held both, and everyone fought over it.

Reaching a Service from your laptop

You are developing and want to hit a ClusterIP Service without any ingress. kubectl port-forward opens a tunnel from your machine to a Service or pod through the API server:

Shell
kubectl port-forward service/notes-api 8080:80# then in another terminal:curl http://localhost:8080/

(Reviewed, not run on a cluster here.) It is a debugging tool, one connection path from you to the cluster, not a way to serve users. It also picks one pod and sticks with it, so do not use it to test load balancing. The other ways to reach a Service from outside are the ones above: NodePort, LoadBalancer, Ingress or a Gateway.

NetworkPolicy: who may talk to whom

By default, all pods can talk to all pods. That is convenient and a little scary. A NetworkPolicy lets you restrict this, like a firewall written in labels. Here is a policy that only allows pods labelled app: notes-api to reach the Postgres pods on port 5432:

YAML
# networkpolicy.yamlapiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:  name: db-only-from-apispec:  podSelector:    matchLabels:      app: postgres  policyTypes:    - Ingress  ingress:    - from:        - podSelector:            matchLabels:              app: notes-api      ports:        - protocol: TCP          port: 5432

Status: reviewed, not run on a cluster here. Passed kubeconform -strict and yamllint.

Two things to remember:

  1. podSelector at the top says which pods the policy protects. Once any policy selects a pod for ingress, traffic to it is denied unless some policy allows it. The from list says who is allowed.
  2. A NetworkPolicy object is only a request. It is enforced by the cluster's network plugin (the CNI). If your CNI does not support policies, you can create the object and nothing happens. This is a real and quiet trap: your database looks protected and is not. The docs say network policies are implemented by the network plugin, so check that yours does.

This is the Kubernetes version of "least privilege" from the Security series: if one pod is compromised, it should not be able to scan and connect to everything else.

What a service mesh is for (and why you probably do not need one yet)

A service mesh puts a small proxy next to each pod (or on each node) and manages the traffic between services. What it can add, in general terms:

  • Encryption between pods (mutual TLS), so internal traffic is not plain text.
  • Retries, timeouts and circuit breaking configured in one place instead of in every service.
  • Traffic splitting, for example send 5 percent of requests to a new version.
  • Detailed metrics and traces for every call between services.

The price is real: more moving parts, more memory and CPU, more to learn and debug, and one more thing that can break at 3 a.m. For a handful of services a mesh is usually too much. Retries and timeouts in your own code, plus NetworkPolicies and a normal Ingress or Gateway, cover most small systems. I did not read any mesh's documentation for this lesson, so treat this section as an overview and read the project docs before you choose one.

A real mini lab you can do without a cluster

If you do not have a cluster, you can still practise the habits of this lesson:

  1. Write the Deployment, Service and Ingress files above.
  2. Run kubeconform -strict -summary -kubernetes-version 1.34.0 *.yaml to check the shapes. It downloads the official schemas on first use.
  3. Break the files on purpose and read the errors. Here are two real ones. This Service has two typos, selecter and targetport:
TEXT
$ kubeconform -strict -summary -kubernetes-version 1.34.0 svc-typo.yamlsvc-typo.yaml - Service bad is invalid: ... at '/spec/ports/0': additional properties 'targetport' not allowed  - at '/spec': additional properties 'selecter' not allowed

Without -strict, the same file passes (I ran both):

TEXT
$ kubeconform -summary -kubernetes-version 1.34.0 svc-typo.yamlSummary: 1 resource found in 1 file - Valid: 1, Invalid: 0, Errors: 0, Skipped: 0

Always use -strict. Without it, unknown fields are allowed, so a misspelled key is silently ignored. That is the same trap as in a real cluster, where a misspelled field can disappear without an error. kubeconform -strict turns on the strict schemas that reject unknown keys.

With a real cluster (a local kind or minikube cluster is enough), the same files can be applied, and then the commands in the next section show you what is going on.

Commands to know (reviewed, not run on a cluster here)

Shell
kubectl get service                         # list Services, see type and cluster IPkubectl get endpointslices                  # see the pod addresses behind each Servicekubectl describe service notes-api          # selector, ports, and Endpoints linekubectl get ingress                         # list Ingresses and their addresskubectl describe ingress notes              # rules, backends, and events from the controllerkubectl run tmp --rm -it --image=busybox:1.36 -- sh   # a throwaway pod to test DNS from inside#   inside it:  nslookup notes-api   and   wget -qO- http://notes-api

The last pair is the most useful test in the whole lesson. Most "networking problems" come down to one of: wrong selector (empty endpoints), wrong targetPort, a pod that is not ready, a DNS name typed in the wrong namespace, or a NetworkPolicy that blocks the call. Testing from inside the cluster with a throwaway pod separates "my Service is broken" from "my Ingress or laptop path is broken". Lesson 8 turns these checks into a full routine.

Try it

  1. A pod in namespace shop wants to call a Service named payments in namespace billing. Write two names that work.
  2. A Service has port: 80 and no targetPort. The container listens on 3000. What happens, and what is the fix?
  3. You created an Ingress and applied it. The cluster has no Ingress controller installed. What traffic is routed?
  4. Why can't a client reach a ClusterIP Service from a laptop outside the cluster, even if the laptop can reach the nodes?
  5. A StatefulSet needs each pod to be reachable by its own DNS name. Which Service type do you create?
  6. You have 3 pods behind a Service, and kubectl describe service shows no endpoints. List three things to check.
  7. Write an Ingress path rule so that /shop and /shop/cart go to Service shop but /shopping does not.

Answers

  1. payments.billing and payments.billing.svc.cluster.local.
  2. The Service sends traffic to pod port 80 (it uses port when targetPort is missing), where nothing listens. Connections are refused. Fix: set targetPort: 3000 (or the named port).
  3. None. An Ingress is only rules. You need a controller to act on them.
  4. The ClusterIP is virtual. It exists only as packet rewrite rules on the cluster's nodes, so it is not routable from outside. Use NodePort, LoadBalancer, Ingress, Gateway, or kubectl port-forward.
  5. A headless Service (clusterIP: None), referenced by the StatefulSet's serviceName.
  6. (a) The Service selector matches the pod labels exactly. (b) The pods are Ready (readiness probe passing). (c) The Service and the pods are in the same namespace. Also check the pods exist at all.
  7. path: /shop with pathType: Prefix. Prefix matches by whole path elements, so /shop and /shop/cart match but /shopping does not.

Interview questions

Why do we need a Service when pods already have IPs? (Must know)

Pod IPs are not stable. When a pod dies, its replacement gets a different IP, and a rollout or a scale-up changes the whole set. A Service gives clients one stable virtual IP and DNS name, and keeps the list of ready pod addresses behind it up to date by watching a label selector. The client talks to notes-api and does not care how many pods exist or where they run.

What are the Service types, and when would you use each? (Must know)

ClusterIP is the default and works only inside the cluster, so I use it for service-to-service calls. NodePort opens a port on every node, which I use for quick tests or as a base for something else. LoadBalancer asks the cloud for an external load balancer, which is good for exposing one thing but costs one load balancer per Service. ExternalName is only a DNS alias to an outside name. And a headless Service, with clusterIP: None, has no virtual IP, and DNS returns the pod addresses, which I use for StatefulSets and client-side balancing.

What is the difference between port, targetPort, nodePort and containerPort? (Must know)

port is where the Service listens, and what clients use. targetPort is the pod port the Service forwards to, and it can be a number or a name. nodePort is a port opened on every node for a NodePort Service, in the 30000 to 32767 range by default. containerPort is the port the app listens on and is mostly documentation. The classic bug is a missing targetPort: Kubernetes then reuses port, and traffic goes to the wrong pod port.

What is the difference between an Ingress and an Ingress controller? (Must know)

An Ingress is a Kubernetes object that holds routing rules: host and path to a Service. An Ingress controller is a program running in the cluster that watches those objects and configures a real proxy or load balancer to match. Without a controller, an Ingress does nothing at all. Also, controllers differ in the annotations they support, so an Ingress is only partly portable.

How does a pod find a Service by name? (Must know)

Through cluster DNS. Every Service gets a name like notes-api.default.svc.cluster.local. Each pod's resolver has a search list with its own namespace first, so the short name notes-api works inside the same namespace, and notes-api.other-namespace works across namespaces. There are also environment variables, but they are set only at pod start, so DNS is the better method.

A Service exists but requests fail, and the endpoints list is empty. What do you check? (Trap)

An empty endpoint list means no ready pod matches the Service. I check, in order, that the selector exactly matches the pod labels (a typo passes every schema check), that pods exist in the same namespace, and that they are Ready, because a failing readiness probe removes a pod from the endpoints. The trap is blaming the network or the Ingress first. If endpoints are empty, nothing else matters yet.

Is a Service a load balancer? How does it balance? (Senior signal)

Yes, but a simple one. For ClusterIP traffic each node's kube-proxy programs packet rules that rewrite the Service IP to one ready pod when a connection opens. There is no central proxy. It balances connections, not requests, and it does not look at load. So with long-lived connections like gRPC, one client may stay on one pod, and new pods get little traffic from old clients. For those I would use client-side balancing or a proxy that understands the protocol.

Ingress is "frozen". What does that mean, and what do you use for a new project? (Senior signal)

The Kubernetes docs say the Ingress API is generally available and will not be removed, but it gets no new development, and the project recommends the Gateway API. Gateway API is a set of add-on resources: GatewayClass, Gateway, HTTPRoute, GRPCRoute. It separates roles (platform team owns the Gateway, app teams own routes), supports things like header matching and traffic weights without controller-specific annotations, and is more portable. For an existing cluster full of Ingress objects I would keep them working and plan a gradual move. For a new design I would check whether my controller supports Gateway API.

What is a headless Service and why would you want one? (Senior signal)

It is a Service with clusterIP: None. It gets no virtual IP, kube-proxy ignores it, and DNS returns the addresses of the ready pods directly. I want one when clients need to know individual pods, for example members of a database cluster, or when the client does its own load balancing. StatefulSets need a headless Service to give each pod a stable DNS name.

Do NetworkPolicies protect me if I create them? (Trap)

Not necessarily. A NetworkPolicy is only an object. The network plugin (CNI) enforces it, and if the plugin does not support policies, nothing is blocked while everything looks configured. By default all pods can reach all pods. So I first confirm my CNI enforces policies, then I write a default-deny rule for sensitive pods and allow only the callers I expect. Then I test it by trying a connection that should fail.

A service must talk to something outside the cluster by a friendly name. What are your options? (Senior signal)

Simplest is a plain DNS name in config. If I want a stable in-cluster name that I can repoint later, I can make an ExternalName Service that returns a CNAME. A Service without a selector plus manual EndpointSlices also works for fixed IPs. I mention the limits of ExternalName: it is only DNS, so there is no port remapping, and with HTTPS the host name may not match the certificate.

When would you add a service mesh? (Senior signal)

When I have many services and a real need for something a mesh gives centrally: mutual TLS between all pods, uniform retries and timeouts, traffic splitting for canary releases, and per-call metrics. I would not add one for a few services. It costs memory and CPU for every proxy and adds operational complexity. First I would try timeouts and retries in code, NetworkPolicies, and a good Ingress or Gateway.

Key takeaways

  • Pod IPs change. A Service is a stable name and virtual IP that finds pods by label selector.
  • Only ready pods are in the EndpointSlice. Empty endpoints mean a selector typo or no ready pods.
  • Default type ClusterIP works inside the cluster only. NodePort adds a port on each node. LoadBalancer asks the cloud for a load balancer. Headless (clusterIP: None) returns pod IPs through DNS.
  • Inside the cluster, use DNS names: service, service.namespace, or the full service.namespace.svc.cluster.local.
  • An Ingress is rules only, and needs an Ingress controller. pathType is required. The Ingress API is frozen, and the Kubernetes project recommends the Gateway API for new work.
  • NetworkPolicy needs a CNI that enforces it. By default every pod can talk to every pod.
  • A service mesh is for large systems with real needs. Do not start with one.
  • A schema check passes a Service with a wrong selector. Use kubeconform -strict, and then test in a real cluster.

What's next

Next: Configuration, Secrets and Storage in Kubernetes, Lesson 6. Your pods can now find each other. Next they need settings, passwords and disk space. You will meet ConfigMaps, Secrets (and why base64 is not encryption), and persistent volumes. Practise now: write the Deployment, Service and Ingress for one of your own apps, and run kubeconform -strict over them.

Published by Jainik · Last updated · How these lessons are made

Keep learning

Related lessons

Part of the study guide

Related series