StatefulSets Explained

Introduction¶

A Deployment treats pods as interchangeable cattle. A StatefulSet gives each pod a stable identity that survives rescheduling — a persistent name, a persistent volume, and a predictable order. That's what clustered databases, message queues, and consensus systems need.

The guarantees¶

Stable network identity¶

Pods are named <statefulset>-0, <statefulset>-1, … (the ordinal). Pod db-1 is always db-1, even after it's deleted and recreated on another node.

With a headless Service (clusterIP: None), each pod gets a stable DNS name:

db-0.db.team-a.svc.cluster.local
db-1.db.team-a.svc.cluster.local

so db-1 can always find db-0 (e.g. to replicate from it). The headless Service is required — the StatefulSet's serviceName field must point at it.

Stable storage¶

volumeClaimTemplates creates one PVC per pod: data-db-0, data-db-1. When db-1 is rescheduled, it re-attaches the same PVC. Deleting the StatefulSet does not delete the PVCs (by default) — your data is safe from a kubectl delete.

Ordered operations¶

• Scale up: db-0, then db-1, then db-2 — each waiting for the previous to be Running and Ready.
• Scale down: reverse order, db-2 first.
• Rolling update: highest ordinal first, down to 0, one at a time, waiting for Ready.

podManagementPolicy: Parallel disables the ordering for start/stop (not updates) if your app doesn't need it.

A StatefulSet¶

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db                 # the headless Service
  replicas: 3
  selector:
    matchLabels: { app: db }
  template:
    metadata:
      labels: { app: db }
    spec:
      containers:
        - name: postgres
          image: postgres:16
          ports: [{ containerPort: 5432, name: pg }]
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata: { name: data }
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: fast
        resources: { requests: { storage: 100Gi } }
---
apiVersion: v1
kind: Service
metadata: { name: db }
spec:
  clusterIP: None
  selector: { app: db }
  ports: [{ port: 5432, name: pg }]

Partitioned rollouts (canary)¶

updateStrategy:
  type: RollingUpdate
  rollingUpdate:
    partition: 2      # only pods with ordinal >= 2 update

Set partition: 2 on a 3-replica set → only db-2 gets the new version. Verify it, then lower to 0 to roll the rest. A manual canary for stateful systems.

When NOT to use a StatefulSet¶

• Your app is actually stateless — use a Deployment; StatefulSet's ordering just slows deploys.
• You need shared (RWX) storage across all pods — that's a Deployment + an RWX PVC, not per-pod volumes.
• The "state" is external (RDS, managed Kafka) — the app tier is stateless.
• You want fast parallel scaling and pods don't need identity — Deployment.

Verification and troubleshooting¶

kubectl get statefulset db
kubectl get pods -l app=db -o wide       # -0, -1, -2 in order
kubectl get pvc -l app=db
kubectl get svc db                       # CLUSTER-IP should be None
kubectl exec db-1 -- nslookup db-0.db

• Pods stuck at db-0 Pending, rest never start — ordered startup: db-0 isn't Ready (failing readiness probe, PVC unbound, image pull). Fix db-0 first. Parallel podManagementPolicy if ordering isn't needed.
• db-1 can't resolve db-0.db — serviceName doesn't match the headless Service name, the Service isn't headless (clusterIP: None), or it has a publishNotReadyAddresses need (peers must find each other before Ready — set publishNotReadyAddresses: true on the Service).
• PVC stuck Pending — StorageClass issues (see Persistent Volumes and Claims). WaitForFirstConsumer is normal until the pod schedules.
• Scaled down then up, data gone — you deleted the PVCs, or the new pod landed in a different zone and couldn't attach the zonal volume. PVCs persist by default; check kubectl get pvc.
• Update stuck — partition is set above 0, or a pod won't become Ready so the rollout won't proceed to the next ordinal. kubectl rollout status statefulset/db.
• kubectl delete statefulset left PVCs and pods — --cascade=orphan was used, or PVCs are retained by design. Delete PVCs explicitly if you mean to.
• Multi-Attach error after a node crash — the old pod's RWO volume is still attached to the dead node; wait for the timeout or force-delete the old pod.

Related tools and reading¶

• On-site: YAML ↔ JSON converter.
• Related posts: Persistent Volumes and Claims, Kubernetes Services and Endpoints.