Image and bundle are different things​

The Open Container Initiative publishes three separate specifications, and it is worth being precise about which does what:

The step between them is the one people miss. A container manager pulls an image, unpacks its layers into a single directory, merges the image's config with any runtime overrides you supplied, and writes the result as config.json. Only then does it hand the directory to a runtime.

image (registry)  →  unpack layers + merge config  →  bundle (on disk)  →  runc  →  process

An image can produce many bundles; a bundle is a single container's complete instructions.

Making one by hand​

This is the fastest way to make the abstraction real. It takes about a minute on any Linux host with Docker and runc installed.

mkdir -p /tmp/bundle/rootfs && cd /tmp/bundle

# 1. Get a root filesystem. `docker export` flattens an image's layers into a tar.
docker export "$(docker create alpine:3.21)" | tar -C rootfs -xf -

# 2. Generate a default config.json
runc spec

ls -la
# config.json  rootfs/

That is the entire bundle. Two entries:

• rootfs/ — the filesystem the container will see as /.
• config.json — everything else.

sudo runc run demo
# / # cat /etc/os-release

No image, no daemon, no registry. A directory and a JSON file.

Inside config.json​

The spec version generated by runc spec is long but every section has a clear purpose. Here is an annotated tour of the parts that matter.

ociVersion and process​

{
  "ociVersion": "1.2.0",
  "process": {
    "terminal": true,
    "user": { "uid": 0, "gid": 0, "additionalGids": [] },
    "args": ["sh"],
    "env": ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "TERM=xterm"],
    "cwd": "/",
    "noNewPrivileges": true,
    "rlimits": [{ "type": "RLIMIT_NOFILE", "hard": 1024, "soft": 1024 }]
  }
}

process.args is the container's PID 1 — this is where a Dockerfile's ENTRYPOINT and CMD end up after being concatenated. noNewPrivileges sets the kernel's no_new_privs bit, which is exactly what Kubernetes' allowPrivilegeEscalation: false produces.

capabilities​

"capabilities": {
  "bounding":    ["CAP_CHOWN", "CAP_NET_BIND_SERVICE", "CAP_KILL"],
  "effective":   ["CAP_CHOWN", "CAP_NET_BIND_SERVICE", "CAP_KILL"],
  "permitted":   ["CAP_CHOWN", "CAP_NET_BIND_SERVICE", "CAP_KILL"],
  "inheritable": [],
  "ambient":     []
}

Linux capabilities split root's power into ~40 separate privileges. A "root" process in a container is normally root with most of them removed — which is why containerised root cannot load kernel modules (CAP_SYS_MODULE) or change the system clock (CAP_SYS_TIME).

The five sets are not interchangeable:

Kubernetes' capabilities.drop: ["ALL"] empties all of these. CAP_NET_BIND_SERVICE is the one people add back, to bind port 80 as a non-root user.

root and hostname​

"root": {
  "path": "rootfs",
  "readonly": false
},
"hostname": "runc"

root.path is relative to the bundle directory. readonly: true is what Kubernetes' readOnlyRootFilesystem: true sets.

mounts​

"mounts": [
  { "destination": "/proc", "type": "proc", "source": "proc" },
  {
    "destination": "/dev",
    "type": "tmpfs",
    "source": "tmpfs",
    "options": ["nosuid", "strictatime", "mode=755", "size=65536k"]
  },
  {
    "destination": "/sys",
    "type": "sysfs",
    "source": "/sys",
    "options": ["nosuid", "noexec", "nodev", "ro"]
  },
  {
    "destination": "/data",
    "type": "bind",
    "source": "/var/lib/myapp",
    "options": ["rbind", "rw"]
  }
]

Every Kubernetes volume mount, every docker -v, every ConfigMap projection ultimately becomes an entry in this array. /proc, /sys, and /dev are mounted fresh inside the container's mount namespace so it sees its own process table and device nodes rather than the host's.

linux.namespaces​

"linux": {
  "namespaces": [
    { "type": "pid" },
    { "type": "network" },
    { "type": "ipc" },
    { "type": "uts" },
    { "type": "mount" },
    { "type": "cgroup" }
  ]
}

This is the heart of it. Each entry with no path means "create a new one." An entry with a path means "join this existing namespace":

{ "type": "network", "path": "/proc/1234/ns/net" }

That single line is how a Kubernetes Pod works. Every container in a Pod joins the same network namespace — the pause container's — which is why they share an IP address and reach each other on localhost. It is also what docker run --network=container:other and kubectl debug --target produce.

linux.resources​

"resources": {
  "memory": { "limit": 268435456 },
  "cpu": { "shares": 1024, "quota": 100000, "period": 100000 },
  "pids": { "limit": 512 }
}

These are written straight into the cgroup filesystem. A Kubernetes limits.memory: 256Mi becomes memory.limit here, and exceeding it is what triggers the kernel OOM killer and the OOMKilled status you see in kubectl describe.

CPU is expressed as a quota per period: 100000/100000 is one full core, 50000/100000 is half a core, and exceeding it results in throttling rather than a kill.

linux.seccomp​

"seccomp": {
  "defaultAction": "SCMP_ACT_ERRNO",
  "architectures": ["SCMP_ARCH_X86_64"],
  "syscalls": [
    {
      "names": ["accept", "read", "write", "openat", "close"],
      "action": "SCMP_ACT_ALLOW"
    }
  ]
}

A syscall allow-list enforced by the kernel. The default action is to deny; listed syscalls are permitted. Docker's default profile blocks around 44 syscalls, which is what Kubernetes' seccompProfile: { type: RuntimeDefault } applies.

linux.uidMappings​

"uidMappings": [
  { "containerID": 0, "hostID": 100000, "size": 65536 }
],
"gidMappings": [
  { "containerID": 0, "hostID": 100000, "size": 65536 }
]

User-namespace remapping: UID 0 inside the container is UID 100000 on the host. Root in the container is an unprivileged user outside it, so a container escape lands on a nobody account. This is what "rootless containers" means, and Kubernetes exposes it as hostUsers: false.

The lifecycle a runtime implements​

The runtime spec defines operations, not just a file format. runc implements each as a subcommand:

sudo runc create demo
sudo runc state demo        # "status": "created"
sudo runc start demo        # now "running"
sudo runc list
sudo runc kill demo TERM
sudo runc delete demo

The split between create and start is what makes hooks possible. Between the two, a createRuntime or createContainer hook can run on the host with the container's namespaces already in place — which is precisely how a CNI plugin attaches a network interface to a container before its process ever executes.

"hooks": {
  "createRuntime": [
    { "path": "/usr/local/bin/setup-network.sh", "args": ["setup-network.sh"] }
  ],
  "poststop": [
    { "path": "/usr/local/bin/cleanup.sh", "args": ["cleanup.sh"] }
  ]
}

Seeing the real thing on a Kubernetes node​

Every running container on a node has a bundle. On a node using containerd:

# List containers in the kubelet's namespace
sudo ctr -n k8s.io containers list

# Dump the actual OCI spec containerd generated for one
sudo ctr -n k8s.io containers info <container-id> | jq '.Spec'

Compare that output against the Pod's YAML and the mapping is direct — securityContext fields appear under process and linux, volume mounts appear in mounts, and resource limits appear under linux.resources. Every Kubernetes field you set is a translation into this document.

Why this is worth knowing​

• It demystifies containers. There is no magic: a JSON file describes namespaces, cgroups, mounts, and a process, and a runtime asks the kernel for exactly that.
• It explains Kubernetes fields. runAsNonRoot, readOnlyRootFilesystem, capabilities.drop, seccompProfile — each one is a line in this document, and knowing which line tells you exactly what it does and does not protect.
• It explains why runtimes are swappable. crun, gVisor, and Kata read the same config.json. They differ only in how they satisfy it — a C implementation, a user-space kernel, or a microVM.
• It makes debugging possible. When a container behaves unexpectedly, dumping its actual spec answers the question far faster than guessing at the layers above.

Further reading​

• Docker vs containerd vs runc — who produces this bundle and who consumes it.
• Kubernetes security basics — the securityContext fields that map onto these sections.
• The OCI runtime specification itself — genuinely readable, and shorter than you expect.

The mental model​

kubectl is a REST client. Every command is an HTTP request to the API server, and you can see exactly which one:

kubectl get pods -v=6            # show the request URLs
kubectl get pods -v=8            # show full request and response bodies

That is genuinely useful when a command behaves unexpectedly — you can see whether the problem is your query or the cluster's answer.

get — what exists​

kubectl get pods
kubectl get pods -A                      # every namespace
kubectl get pods -o wide                 # + node, pod IP, nominated node
kubectl get pod web-abc -o yaml          # the full object as stored
kubectl get deploy,svc,ing -l app=web    # several kinds at once

The flags worth memorising​

# Watch changes as they happen
kubectl get pods -w

# Filter server-side by field (cheaper than grep, and exact)
kubectl get pods --field-selector=status.phase=Running
kubectl get events --field-selector type=Warning

# Sort by anything in the object
kubectl get pods --sort-by=.status.containerStatuses[0].restartCount
kubectl get events --sort-by=.lastTimestamp

# Pull out exactly the columns you want
kubectl get pods -o custom-columns=\
'NAME:.metadata.name,NODE:.spec.nodeName,IMAGE:.spec.containers[*].image'

# JSONPath for scripting
kubectl get pods -o jsonpath='{.items[*].spec.nodeName}'
kubectl get svc web -o jsonpath='{.spec.clusterIP}'

kubectl api-resources is the command that makes the rest discoverable — it lists every kind the cluster knows about, along with its short name and whether it is namespaced.

kubectl api-resources | head -20
kubectl explain deployment.spec.strategy.rollingUpdate    # inline schema docs

describe — why it is in that state​

kubectl describe pod web-abc
kubectl describe node worker-1
kubectl describe svc web

describe is not just a prettier get. It joins the object with related state — most importantly its Events, which appear at the bottom and are where the actual explanation lives:

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Warning  Failed     2m    kubelet            Error: ImagePullBackOff
  Warning  Unhealthy  1m    kubelet            Readiness probe failed: connection refused

Always read the bottom of describe first. It answers "why" more often than the logs do.

# Two high-value describe habits
kubectl describe pod web-abc | sed -n '/Events:/,$p'         # events only
kubectl describe node worker-1 | sed -n '/Allocated/,$p'     # what is claimed on a node

logs — what the application said​

kubectl logs web-abc
kubectl logs web-abc -f                  # follow
kubectl logs web-abc --tail=100
kubectl logs web-abc --since=15m
kubectl logs web-abc -c sidecar          # a specific container in the Pod

The two flags people miss​

# The container that CRASHED, not the one that just started
kubectl logs web-abc --previous

# Every Pod behind a label, with the pod name prefixed on each line
kubectl logs -l app=web --all-containers --prefix --tail=50

--previous is the single most important debugging flag in kubectl. On a CrashLoopBackOff Pod, plain kubectl logs shows a container that has been alive for two seconds and has nothing to say.

# Timestamps are not on by default
kubectl logs web-abc --timestamps --since-time='2026-05-18T10:00:00Z'

exec — get inside​

kubectl exec web-abc -- env | sort
kubectl exec -it web-abc -- sh
kubectl exec -it web-abc -c sidecar -- bash

The -- matters: everything after it is passed to the container rather than parsed by kubectl. Without it, kubectl exec web-abc ls -la tries to interpret -la as a kubectl flag.

# Useful one-liners from inside a Pod
kubectl exec web-abc -- cat /etc/resolv.conf
kubectl exec web-abc -- nslookup db.default.svc.cluster.local
kubectl exec web-abc -- wget -qO- --timeout=2 localhost:8080/healthz

When the image has no shell — distroless, scratch — exec cannot help. Use an ephemeral debug container instead, which joins the running Pod's namespaces:

kubectl debug -it web-abc --image=nicolaka/netshoot --target=app

port-forward — reach it from your laptop​

kubectl port-forward pod/web-abc 8080:8080
kubectl port-forward svc/web 8080:80        # picks one backing Pod
kubectl port-forward deploy/web 8080:8080

# Bind on all interfaces (careful on shared networks)
kubectl port-forward --address 0.0.0.0 svc/web 8080:80

This is a tunnel through the API server, so it works even when the Service is ClusterIP-only and the cluster is behind a bastion. It is the fastest way to reach a database or an admin UI you would never expose.

Note that forwarding to a Service still selects one Pod and stays with it — it is not load balanced. For testing balancing behaviour, use a NodePort or an Ingress.

apply — declare the desired state​

kubectl apply -f deployment.yaml
kubectl apply -f k8s/                    # a whole directory
kubectl apply -k overlays/prod           # kustomize
kubectl apply -f https://example.com/manifest.yaml

Before you apply​

# Validate against the server's schema without changing anything
kubectl apply -f deployment.yaml --dry-run=server

# See exactly what would change on the live object
kubectl diff -f deployment.yaml

kubectl diff is underused and excellent — it is git diff for the cluster, and it catches the "someone edited this by hand in production" case immediately.

apply vs create vs replace. create fails if the object exists. replace overwrites the whole object, dropping fields you did not send. apply performs a three-way merge between your manifest, the live object, and the last-applied configuration — which is why it is the only one safe to run repeatedly.

Rollouts​

kubectl rollout status deployment/web --timeout=120s
kubectl rollout history deployment/web
kubectl rollout history deployment/web --revision=3
kubectl rollout undo deployment/web
kubectl rollout undo deployment/web --to-revision=2
kubectl rollout restart deployment/web     # recreate every Pod, no spec change

rollout restart is the clean way to force a restart — it patches an annotation on the Pod template, which triggers a normal rolling update. Deleting Pods by hand skips the surge and availability guarantees.

Quick reference​

Make it faster​

# Completion and a short alias that keeps completion working
source <(kubectl completion bash)
alias k=kubectl
complete -o default -F __start_kubectl k

# Stop typing -n every time
alias kn='kubectl config set-context --current --namespace'

Two tools that pay for themselves in a week:

• kubectx / kubens — switch cluster and namespace in one word.
• k9s — a terminal UI over the cluster. Logs, exec, describe, and delete without typing a command; it replaces most of the get/describe loop.

The habits that matter​

1. Read the Events at the bottom of describe before anything else.
2. Use --previous on any Pod that has restarted.
3. Run kubectl diff before kubectl apply on a cluster you care about.
4. Use --dry-run=server to validate manifests in CI.
5. Set the namespace in your context instead of typing -n two hundred times a day.

Further reading​

• The kubectl cheat sheet — the full searchable reference on this site.
• Debugging pod failures — these commands applied to a specific problem.
• The official kubectl reference.

Every command below has been written to run as-is.

Install​

Prerequisites: Docker running, and at least 4 GB of memory available to it (8 GB is comfortable for three nodes).

# macOS
brew install kind kubectl

# Linux (amd64)
[ "$(uname -m)" = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.30.0/kind-linux-amd64
chmod +x ./kind && sudo mv ./kind /usr/local/bin/kind

# Windows
winget install Kubernetes.kind

kind version
docker info | grep -E 'Total Memory|Server Version'

A single node is one command​

kind create cluster --name quickstart
kubectl cluster-info --context kind-quickstart
kind delete cluster --name quickstart

Useful for a smoke test. It teaches you nothing about scheduling, because there is only one node — and it is a control-plane node carrying a taint, which kind removes so your Pods can land somewhere.

The multi-node cluster​

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: nativehub

nodes:
  - role: control-plane
    kubeadmConfigPatches:
      - |
        kind: InitConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "ingress-ready=true"
    extraPortMappings:
      - containerPort: 80
        hostPort: 80
        protocol: TCP
      - containerPort: 443
        hostPort: 443
        protocol: TCP

  - role: worker
    labels:
      topology.kubernetes.io/zone: zone-a
      node-type: general

  - role: worker
    labels:
      topology.kubernetes.io/zone: zone-b
      node-type: general

kind create cluster --config kind-config.yaml

kubectl get nodes -o wide

NAME                      STATUS   ROLES           VERSION
nativehub-control-plane   Ready    control-plane   v1.34.0
nativehub-worker          Ready    <none>          v1.34.0
nativehub-worker2         Ready    <none>          v1.34.0

Each node is a Docker container — this is not a metaphor:

docker ps --filter "name=nativehub" --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"
docker exec -it nativehub-worker crictl ps

Note crictl, not docker. Inside the node, containerd is the runtime; there is no Docker daemon in there.

Two details in that config do real work:

• extraPortMappings publishes ports from the control-plane container to your host, which is how browser traffic on localhost:80 reaches an Ingress controller inside the cluster.
• node-labels: ingress-ready=true marks that node so the ingress-nginx manifest — which includes a matching nodeSelector — schedules there.

Pin the Kubernetes version​

kind's default node image tracks the kind release. To match a specific cluster version, pass a digest-pinned image:

kind create cluster --config kind-config.yaml \
  --image kindest/node:v1.34.0

Each kind release supports a specific set of node images — check the release notes rather than guessing, since an unsupported combination fails during kubeadm init with an unhelpful timeout.

Install an ingress controller​

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.0/deploy/static/provider/kind/deploy.yaml

kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=180s

That manifest is the kind-specific variant: it uses hostPort and the ingress-ready node selector, which is what makes the extraPortMappings above line up.

Deploy something and route to it​

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 4
  selector:
    matchLabels: { app: web }
  template:
    metadata:
      labels: { app: web }
    spec:
      # Spread replicas across zones so a "zone" outage cannot take everything.
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: { app: web }
      containers:
        - name: echo
          image: ealen/echo-server:0.9.2
          ports:
            - containerPort: 80
          resources:
            requests: { cpu: 25m, memory: 32Mi }
            limits: { memory: 64Mi }
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector: { app: web }
  ports:
    - port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
spec:
  ingressClassName: nginx
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 80

kubectl apply -f demo.yaml
kubectl rollout status deployment/web

curl -s localhost/ | jq '.host, .request.headers."x-forwarded-for"'

Check where the scheduler put things:

kubectl get pods -l app=web -o custom-columns=\
'NAME:.metadata.name,NODE:.spec.nodeName,IP:.status.podIP'

With topologySpreadConstraints, the four replicas are split evenly across the two labelled workers. That is genuinely useful to see — it is the same mechanism that keeps production workloads spread across availability zones.

Load local images without a registry​

The most common kind papercut: you build an image locally, reference it in a manifest, and the Pod sits in ErrImagePull. The nodes are separate containers with their own image stores — your host's Docker images are not visible to them.

docker build -t myapp:dev .
kind load docker-image myapp:dev --name nativehub

# Verify it landed
docker exec -it nativehub-worker crictl images | grep myapp

Set imagePullPolicy: IfNotPresent in the manifest. With the default Always behaviour for untagged or :latest images, the kubelet will try to pull from Docker Hub and fail anyway.

A local registry (the better workflow)​

kind load copies the image into every node on every build, which gets slow. A local registry the cluster can pull from is faster and closer to reality:

#!/usr/bin/env bash
set -euo pipefail

reg_name='kind-registry'
reg_port='5001'

# 1. Start a registry container if it is not already running
if [ "$(docker inspect -f '{{.State.Running}}' "${reg_name}" 2>/dev/null || true)" != 'true' ]; then
  docker run -d --restart=always -p "127.0.0.1:${reg_port}:5000" \
    --network bridge --name "${reg_name}" registry:2
fi

# 2. Create the cluster, telling containerd where to find the mirror
cat <<EOF | kind create cluster --name nativehub --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
  - |-
    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = "/etc/containerd/certs.d"
nodes:
  - role: control-plane
  - role: worker
  - role: worker
EOF

# 3. Point each node at the registry
REGISTRY_DIR="/etc/containerd/certs.d/localhost:${reg_port}"
for node in $(kind get nodes --name nativehub); do
  docker exec "${node}" mkdir -p "${REGISTRY_DIR}"
  cat <<EOF | docker exec -i "${node}" cp /dev/stdin "${REGISTRY_DIR}/hosts.toml"
[host."http://${reg_name}:5000"]
EOF
done

# 4. Join the registry to kind's network so the nodes can reach it by name
docker network connect "kind" "${reg_name}" 2>/dev/null || true

chmod +x setup-registry.sh && ./setup-registry.sh

docker build -t localhost:5001/myapp:dev .
docker push localhost:5001/myapp:dev
# Reference it in your manifests as localhost:5001/myapp:dev

Simulate a node failure​

This is the best reason to run multiple nodes locally.

# Watch in one terminal
kubectl get pods -l app=web -o wide --watch

# Kill a node in another
docker stop nativehub-worker2

Within about 40 seconds the node controller marks the node NotReady; after the eviction timeout (five minutes by default) its Pods are marked for deletion and the ReplicaSet schedules replacements on the surviving worker. Bring it back:

docker start nativehub-worker2
kubectl get nodes

You can also practise a graceful drain, which is what a real node upgrade does:

kubectl drain nativehub-worker --ignore-daemonsets --delete-emptydir-data
kubectl get pods -o wide          # everything moved off
kubectl uncordon nativehub-worker

Everyday kind commands​

When kind is not the right tool​

kind is excellent for CI, controller development, and learning. Its limits are real:

• No cloud LoadBalancer. type: LoadBalancer Services stay <pending> unless you install MetalLB or run cloud-provider-kind.
• No persistent storage beyond the local-path provisioner, which is ReadWriteOnce and node-local — you cannot exercise real CSI behaviour.
• Nodes share the host kernel, so kernel-level differences from your production nodes will not show up.
• Everything is ephemeral by design. Deleting the cluster deletes the data.

For a persistent single-node environment, minikube or k3d suit better. For anything involving cloud integrations, nothing substitutes for a real cluster.

Clean up​

kind delete cluster --name nativehub
docker stop kind-registry && docker rm kind-registry

Further reading​

• Deploy your first app to Kubernetes — the natural next step.
• Kubernetes networking — what the Ingress above is doing.
• The official kind documentation.

The rule everything is built on​

A conforming Kubernetes network must satisfy three requirements:

1. Every Pod has its own unique IP address.
2. Any Pod can reach any other Pod by IP, across nodes, without NAT.
3. Agents on a node can reach every Pod on that node.

There is no port mapping between Pods. If your container listens on :8080, another Pod connects to <pod-ip>:8080 directly. This is deliberately unlike Docker's default bridge networking, where you publish ports to the host and everything talks through NAT.

Kubernetes itself does not implement this. A CNI plugin — Calico, Cilium, Flannel, or your cloud provider's VPC-native plugin — does. This is why a fresh cluster with no CNI installed leaves every Pod Pending and every node reporting NetworkReady=false.

Pod IPs are worthless to you​

kubectl get pods -o wide

NAME                   READY   STATUS    IP            NODE
web-7d4b9c6f85-2xk4l   1/1     Running   10.244.1.14   worker-1
web-7d4b9c6f85-9mfqz   1/1     Running   10.244.2.7    worker-2

Those addresses are real and reachable — and completely disposable. Delete a Pod and its IP is gone forever; the replacement gets a new one. Any configuration containing a Pod IP is already wrong.

This impermanence is the entire reason Services exist.

Services: a stable name for a moving target​

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  type: ClusterIP
  selector:
    app: web # matches POD labels
  ports:
    - name: http
      port: 80 # the port the Service answers on
      targetPort: 8080 # the port on the Pod

kubectl apply -f service.yaml
kubectl get svc web

NAME   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)
web    ClusterIP   10.96.171.42   <none>        80/TCP

10.96.171.42 is stable for the Service's whole lifetime. Behind it, the EndpointSlice controller watches for Pods matching the selector and keeps a list of the ready ones:

kubectl get endpointslices -l kubernetes.io/service-name=web -o yaml | grep -A2 addresses

Two consequences worth internalising:

• The selector matches Pod labels, not the Deployment's labels. They are often different, and this is the number one cause of a Service that resolves but connects to nothing.
• Only ready Pods are listed. If your readiness probe is failing, the endpoint list is empty and connections are refused — the Service is working exactly as designed.

The four Service types​

They are cumulative. A LoadBalancer Service still has a NodePort and still has a ClusterIP; the cloud load balancer simply forwards to that NodePort on each node.

ClusterIP — the default​

Internal only. Use it for everything that is not deliberately exposed: databases, internal APIs, caches. Practically every Service in a cluster should be this type.

NodePort — mostly for development​

spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 8080
      nodePort: 30080 # optional; auto-assigned from 30000-32767 if omitted

Every node now listens on 30080 and forwards to the Service, whether or not that node runs a backing Pod. It is useful with kind or minikube; in production it means high, non-standard ports and no TLS termination.

LoadBalancer — the production door​

spec:
  type: LoadBalancer

The cloud-controller-manager provisions a real load balancer and writes its address into status.loadBalancer. The catch is cost and count: one cloud load balancer per Service. Ten HTTP services means ten load balancers and ten bills.

That is what Ingress and the Gateway API exist to solve — a single load balancer fronting an in-cluster proxy that routes by hostname and path.

There is one setting on NodePort and LoadBalancer Services worth knowing: externalTrafficPolicy. The default, Cluster, lets any node forward to a Pod on any other node — even balancing, but the extra hop SNATs the packet and your application sees the node's IP instead of the real client's. Setting it to Local forwards only to Pods on the receiving node, preserving the client IP at the cost of uneven balancing when Pods are unevenly spread.

How a ClusterIP actually works​

This is the part that surprises people: nothing is listening on the ClusterIP. No interface has that address. Nothing answers ARP for it. You cannot ping it meaningfully.

It exists only as a set of kernel rules:

1. kube-proxy runs on every node and watches Services and EndpointSlices from the API server.
2. In iptables mode (the default) it writes NAT rules: packets destined for 10.96.171.42:80 are DNAT'd to one of the ready backing Pod IPs, chosen at random with equal probability.
3. In IPVS mode it programs kernel IPVS virtual servers instead — better scaling past a few thousand Services, and real balancing algorithms.

The rewrite happens in the kernel on the sending node, before the packet ever leaves. kube-proxy is not in the data path; it only maintains rules. That is why kube-proxy can be restarted without dropping connections.

# On a node: the rules kube-proxy installed
sudo iptables -t nat -L KUBE-SERVICES -n | head -20

Pod-to-pod, step by step​

A Pod in namespace frontend calls http://web.backend.svc.cluster.local:

1. DNS. The request goes to CoreDNS, which returns the Service's ClusterIP — say 10.96.171.42.
2. Connect. The Pod opens a TCP connection to 10.96.171.42:80.
3. DNAT. As the packet leaves the Pod's network namespace it hits the node's NAT table. kube-proxy's rules rewrite the destination to a ready Pod, 10.244.2.7:8080.
4. Route. The CNI plugin delivers it — over the local bridge if that Pod is on this node, or across the node network (VXLAN tunnel, BGP route, or native VPC routing) if it is not.
5. Reply. conntrack un-NATs the response, so the client sees a reply from the address it dialled.

The Pod never knew it was load balanced. It dialled one address and got an answer.

DNS inside the cluster​

CoreDNS runs as a Deployment in kube-system, exposed by its own ClusterIP Service. The kubelet writes every Pod's /etc/resolv.conf to point at it:

nameserver 10.96.0.10
search frontend.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The canonical record is <service>.<namespace>.svc.cluster.local, and the search path lets you abbreviate:

Cross-namespace calls need at least <service>.<namespace>. A bare web will only ever find a Service in the caller's own namespace.

The ndots:5 gotcha. Any name with fewer than five dots is tried against every search suffix before being tried as written. Looking up api.github.com (two dots) therefore issues api.github.com.frontend.svc.cluster.local, then ...svc.cluster.local, then ...cluster.local, and only then the real query — four wasted round trips per lookup. For hot external calls, use a trailing dot (api.github.com.) to force an absolute lookup, or set a lower ndots in the Pod's dnsConfig.

Headless Services​

Setting clusterIP: None allocates no virtual IP and installs no proxy rules. DNS then returns the A records of the individual Pods:

$ nslookup db.default.svc.cluster.local
Address: 10.244.1.21
Address: 10.244.2.33
Address: 10.244.3.9

This is what StatefulSets use. Combined with a StatefulSet's stable names, each replica gets its own resolvable record — db-0.db.default.svc.cluster.local — which is exactly what a database client needs to distinguish a primary from a replica.

Debugging: three commands in order​

# 1. Does the Service have any endpoints?
kubectl get endpointslices -l kubernetes.io/service-name=web
# Empty => selector mismatch or failing readiness probe. Stop here and fix it.

# 2. Does DNS resolve?
kubectl run netshoot --rm -it --restart=Never --image=nicolaka/netshoot -- \
  nslookup web.default.svc.cluster.local

# 3. Does the Pod itself answer, bypassing the Service?
kubectl exec -it netshoot -- curl -sv 10.244.2.7:8080
# Works => Service/kube-proxy problem. Fails => application or CNI problem.

That third step is the one that saves the most time: it splits "the Service is misconfigured" from "the application is not listening" in a single command.

The five ideas​

1. Every Pod gets a real, routable IP — and it is disposable.
2. A Service is a stable virtual IP over a label-selected set of ready Pods.
3. That virtual IP is only kernel NAT rules, maintained by kube-proxy. Nothing listens on it.
4. CoreDNS maps service.namespace.svc.cluster.local to that virtual IP, and ndots:5 makes short names cheap and external names expensive.
5. Service types are cumulative: LoadBalancer contains NodePort contains ClusterIP.

Further reading​

• Kubernetes networking — Ingress, Gateway API, and NetworkPolicy in more depth.
• Debugging pod failures — when it is not the network.
• The official Services, Load Balancing, and Networking documentation.

The stack, top to bottom​

docker CLI                    what you type
    │  REST over a Unix socket
    ▼
dockerd                       image build, networking, volumes, Compose
    │  gRPC
    ▼
containerd                    image pull/store, container lifecycle, snapshots
    │  spawns one shim per container
    ▼
containerd-shim-runc-v2       keeps the container alive independently of containerd
    │  exec
    ▼
runc                          sets up namespaces + cgroups, then execs your process
    │  syscalls: clone(), unshare(), setns(), pivot_root(), write to cgroupfs
    ▼
Linux kernel                  the only thing that actually "creates" a container

Each arrow is a real, documented interface, not an implementation detail.

runc: the low-level runtime​

runc is a single static binary that implements the OCI Runtime Specification. Its entire job is: given a directory on disk containing a root filesystem and a config.json, create the container the config describes and run the process inside it.

It does not know what an image is. It cannot pull from a registry. It has no notion of tags, networks, or volumes. It performs, roughly:

1. clone() / unshare() with the requested namespace flags — new PID, mount, network, UTS, IPC, and optionally user namespace.
2. Writes the resource limits into the cgroup filesystem.
3. Sets up mounts: /proc, /sys, /dev, and anything the config lists.
4. pivot_root() into the bundle's rootfs.
5. Drops capabilities, applies the seccomp filter, applies the AppArmor or SELinux label.
6. execve() the process from process.args.

You can drive it by hand, which is the fastest way to make the abstraction concrete:

mkdir -p /tmp/mycontainer/rootfs && cd /tmp/mycontainer

# Borrow a root filesystem from an existing image
docker export $(docker create alpine:3.21) | tar -C rootfs -xf -

# Generate a default config.json
runc spec

# Run it — no Docker, no containerd, no daemon
sudo runc run demo

You are now in a shell inside a container that no container manager knows about. That is all a container fundamentally is.

Alternative low-level runtimes implement the same spec and slot into the same position: crun (written in C, faster startup, lower memory), gVisor/runsc (a user-space kernel that intercepts syscalls), and Kata Containers (each container in a lightweight VM). Because the interface is specified, swapping them is a configuration change, not a migration.

containerd: the daemon​

containerd is a long-running daemon and a CNCF graduated project. It handles everything runc deliberately does not:

• Pulling images from registries and verifying their digests
• Unpacking image layers into snapshots (overlayfs, btrfs, devmapper)
• Managing the container lifecycle: create, start, stop, delete, and exec
• Attaching stdio, collecting exit codes, and reporting events
• Namespacing its own state, so Docker and Kubernetes can share one containerd without collisions

When containerd starts a container it does not fork runc as a child. It launches a shim (containerd-shim-runc-v2), one per container, and the shim invokes runc. This indirection is the reason containerd can be restarted or upgraded without killing running containers: the shim is the container's parent and keeps holding its stdio and exit status, so containerd can reconnect afterwards.

# ctr is containerd's low-level debug CLI — not a Docker replacement
sudo ctr namespaces list
sudo ctr -n k8s.io images list | head
sudo ctr -n k8s.io containers list

# nerdctl is the Docker-compatible CLI for containerd
nerdctl run -d -p 8080:80 nginx:1.27-alpine

Note the k8s.io namespace. On a Kubernetes node, that is where the kubelet's containers live — which is exactly why docker ps on such a node shows nothing.

Docker: the developer toolkit​

Docker is the layer above containerd, and it is where the things developers care about actually live:

• docker build and BuildKit — Dockerfiles, layer caching, multi-stage builds
• Networking: bridge networks, port publishing, embedded DNS
• Volume management
• Docker Compose
• The user-facing CLI and REST API

Docker donated containerd to the CNCF in 2017 and then adopted it internally. Since Docker 18.09, docker run ends up as a gRPC call to containerd, which spawns a shim, which execs runc.

# The chain, visible in the process tree
docker run -d --name demo nginx:1.27-alpine
ps -ef --forest | grep -E 'containerd|nginx' | head

Where the CRI fits​

Kubernetes never wanted to talk to a specific vendor's daemon, so it defines the Container Runtime Interface — a gRPC API the kubelet uses for image and container operations. Any runtime implementing CRI can back a Kubernetes node.

• containerd implements CRI natively via its built-in CRI plugin.
• CRI-O is a runtime built solely for Kubernetes: CRI in, OCI runtime out, nothing else.
• Docker never implemented CRI. The kubelet used a built-in adapter called dockershim to translate.

That adapter is what was removed in Kubernetes 1.24. Nothing about "Docker images" was deprecated — they are OCI images, and every CRI runtime runs them unchanged. What went away was the kubelet's special-case translation layer for talking to dockerd. Since Docker already sat on top of containerd, removing it from the node path simply deleted a hop:

Before 1.24:  kubelet → dockershim → dockerd → containerd → shim → runc
After  1.24:  kubelet → CRI ───────────────→ containerd → shim → runc

crictl is the debugging CLI for whatever CRI runtime a node uses:

sudo crictl ps
sudo crictl images
sudo crictl logs <container-id>
sudo crictl inspect <container-id>

The three specifications​

The reason all of this interoperates is that the Open Container Initiative standardised the boundaries:

An image built by docker build, pushed to GHCR, pulled by containerd, and run by crun crosses four different implementations without anyone negotiating anything, because each hand-off is specified.

So which one do I use?​

The one-sentence summary​

Docker is a developer toolkit that delegates to containerd, a daemon that manages images and container lifecycle and delegates to runc, a binary that asks the kernel for namespaces and cgroups and then execs your process. Kubernetes talks to containerd (or CRI-O) through the CRI and skips Docker on the node entirely.

Further reading​

• What is an OCI bundle? — a close look at config.json.
• Docker fundamentals — the layer model and Dockerfile practice.
• The OCI runtime spec and containerd architecture docs.

A Pod is the smallest deployable unit​

You cannot schedule a container. The smallest object Kubernetes will place on a node is a Pod: one or more containers that share a network namespace and can share volumes.

Sharing the network namespace has a concrete consequence. Every container in a Pod sees the same localhost and the same IP address, and they cannot bind the same port — two containers both listening on :8080 in one Pod is a port conflict, exactly as it would be on a single host.

apiVersion: v1
kind: Pod
metadata:
  name: web
  labels:
    app: web
spec:
  containers:
    - name: nginx
      image: nginxinc/nginx-unprivileged:1.27-alpine
      ports:
        - containerPort: 8080
      resources:
        requests:
          cpu: 50m
          memory: 64Mi
        limits:
          memory: 128Mi

kubectl apply -f pod.yaml
kubectl get pod web -o wide

This works. It is also a trap.

Why a bare Pod is not enough​

Delete it and see what happens:

kubectl delete pod web
kubectl get pods
# No resources found in default namespace.

Nothing brought it back, because nothing owns it. A Pod is a record of "this container should run on this node." When the Pod goes away — you deleted it, the node was drained, the node caught fire — that record is simply gone.

A bare Pod gives you:

• ❌ No rescheduling after a node failure
• ❌ No way to run more than one copy without copy-pasting YAML
• ❌ No rolling update — changing the image means deleting and recreating, with downtime
• ❌ No rollback

The chain: Deployment → ReplicaSet → Pod​

The fix is a controller. The most common one is a Deployment, and the crucial detail is that a Deployment does not manage Pods directly:

Deployment  "I want 3 replicas of this Pod template"
    │  owns
    ▼
ReplicaSet  "I own exactly 3 Pods matching this exact template"
    │  owns
    ▼
Pod  Pod  Pod

Each level has one job:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app: web # immutable after creation
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1 # at most 4 Pods during a rollout
      maxUnavailable: 0 # never fewer than 3 ready
  template:
    metadata:
      labels:
        app: web # must satisfy spec.selector
    spec:
      containers:
        - name: nginx
          image: nginxinc/nginx-unprivileged:1.27-alpine
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /
              port: 8080
            initialDelaySeconds: 2
            periodSeconds: 5

Now the ownership is visible in the cluster:

kubectl apply -f deployment.yaml
kubectl get deploy,rs,pod -l app=web

NAME                  READY   UP-TO-DATE   AVAILABLE
deployment.apps/web   3/3     3            3

NAME                             DESIRED   CURRENT   READY
replicaset.apps/web-7d4b9c6f85   3         3         3

NAME                       READY   STATUS    RESTARTS
pod/web-7d4b9c6f85-2xk4l   1/1     Running   0
pod/web-7d4b9c6f85-9mfqz   1/1     Running   0
pod/web-7d4b9c6f85-hd7vw   1/1     Running   0

The Pod name is not random: web-7d4b9c6f85-2xk4l is <deployment>-<template-hash>-<random>. That middle segment is a hash of the Pod template, and the ReplicaSet carries the same one. It is also injected as a label, pod-template-hash, which is how each ReplicaSet claims only its Pods without the templates' selectors colliding.

# Ownership is recorded explicitly on the child object
kubectl get pod -l app=web -o jsonpath='{.items[0].metadata.ownerReferences[0]}' | jq
# { "apiVersion": "apps/v1", "kind": "ReplicaSet", "name": "web-7d4b9c6f85", ... }

Self-healing, demonstrated​

kubectl delete pod -l app=web --field-selector=status.phase=Running --wait=false
kubectl get pods -l app=web

A replacement appears within seconds. Nothing special-cased "a Pod died". The ReplicaSet controller runs the same loop it always runs: desired 3, observed 2, create 1. The identical loop is what recovers from a node failure — the node controller marks the Pods on a dead node as terminating, the count drops, and replacements are scheduled elsewhere.

Why the extra ReplicaSet layer earns its keep​

Change the image:

kubectl set image deployment/web nginx=nginxinc/nginx-unprivileged:1.27.3-alpine
kubectl rollout status deployment/web
kubectl get rs -l app=web

NAME             DESIRED   CURRENT   READY
web-7d4b9c6f85   0         0         0      <- old template, scaled to zero
web-5f8c2d9b47   3         3         3      <- new template

The Deployment created a second ReplicaSet for the new template and moved replicas across one at a time, respecting maxSurge and maxUnavailable, and waiting for each new Pod to pass its readiness probe before retiring an old one.

The old ReplicaSet was kept at zero rather than deleted. That is the entire rollback mechanism:

kubectl rollout history deployment/web
kubectl rollout undo deployment/web

Rolling back scales the previous ReplicaSet back up and the current one down. No image rebuild, no registry pull of something new, no YAML archaeology — the previous desired state is still sitting in the cluster. revisionHistoryLimit controls how many of these zero-scaled ReplicaSets are kept; set it to 0 and you have nothing to roll back to.

The two mistakes everyone makes once​

The selector must match the template labels. spec.selector.matchLabels and spec.template.metadata.labels are separate fields and it is entirely possible to write a Deployment where they disagree. The API server rejects it — selector does not match template labels — but the error is easy to misread as a YAML problem.

The selector is immutable. You cannot change spec.selector after creation. If you need different labels, create a new Deployment and delete the old one. This is why version information must never appear in the selector: the first upgrade that changes it will fail with field is immutable.

When a Deployment is the wrong choice​

A Deployment's Pods are deliberately anonymous — web-7d4b9c6f85-2xk4l means nothing, and Pod 2 is not distinguishable from Pod 3. When identity matters (db-0 must always reattach the same disk), you want a StatefulSet instead.

The takeaway​

• A Pod is the unit of scheduling. Alone, it is unmanaged and disappears for good.
• A ReplicaSet keeps N copies of one exact Pod template alive.
• A Deployment manages ReplicaSets, giving you versioned rollouts and one-command rollback.

Create Pods directly only for throwaway debugging (kubectl run --rm -it). For anything that should survive the night, describe the desired state and let a controller converge on it.

Further reading​

• Kubernetes fundamentals — the wider object model.
• Deploy your first app to Kubernetes — this post, hands on.
• Deployments in the official documentation.