Persistent Volumes and Claims, Explained

Introduction¶

Pods are stateless by design; when they need durable storage, Kubernetes separates the request (PersistentVolumeClaim) from the resource (PersistentVolume), with a StorageClass automating the link.

The three objects¶

• StorageClass — names a provisioner (EBS, GCE PD, Ceph RBD, NFS, local-path) and parameters (disk type, filesystem, encryption). The "how to make storage".
• PersistentVolumeClaim — "I want 20Gi, ReadWriteOnce, from class fast". Namespaced. What you put in your app manifests.
• PersistentVolume — the actual volume, cluster-scoped. Either pre-created by an admin (static) or created on demand by the provisioner (dynamic).

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: fast
  resources:
    requests:
      storage: 20Gi

# In the pod / StatefulSet
volumes:
  - name: data
    persistentVolumeClaim:
      claimName: data

StatefulSets use volumeClaimTemplates to give each replica its own PVC (data-web-0, data-web-1).

Access modes¶

Block storage (EBS, PD, RBD) is RWO only. For RWX you need a file storage backend: NFS, CephFS, EFS, Azure Files. Asking for RWX from an EBS StorageClass → PVC stuck Pending forever.

Binding modes and reclaim policy¶

• volumeBindingMode: WaitForFirstConsumer — don't provision until a pod using the PVC is scheduled, so the volume lands in the same zone as the pod. Use this for zonal block storage; Immediate can put the disk in zone A and the pod in zone B.
• reclaimPolicy: Delete (default for dynamic) — delete the underlying disk when the PVC is deleted. Retain — keep it (manual cleanup, but safe for data you can't lose).

Expanding a volume¶

# StorageClass must have:
allowVolumeExpansion: true

kubectl patch pvc data -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'

The volume grows online for most drivers; the filesystem resize may need a pod restart. You cannot shrink.

Verification and troubleshooting¶

kubectl get pvc
kubectl get pv
kubectl describe pvc data              # Events explain a Pending claim
kubectl get storageclass
kubectl get events --field-selector involvedObject.name=data

• PVC stuck Pending —
• no default StorageClass and none specified (kubectl get sc — one should be (default));
• requested an access mode the provisioner can't do (RWX on block storage);
• WaitForFirstConsumer and no pod is scheduled yet (this is normal until a consumer appears);
• provisioner CSI driver not installed / crashing (kubectl get pods -n kube-system | grep csi);
• quota / cloud limit on disks reached.
• Pod stuck ContainerCreating, "FailedMount" / "Multi-Attach error" — an RWO volume is still attached to another node because the old pod didn't terminate cleanly (node crash). Wait for the attach-detach controller (~6 min), or force-delete the old pod.
• Data gone after PVC recreated — reclaimPolicy: Delete removed the disk. Use Retain for stateful data and back it up.
• resize didn't take effect — driver needs a pod restart for the fs resize, or allowVolumeExpansion is false on the class.
• Wrong zone — Immediate binding. Recreate the StorageClass with WaitForFirstConsumer.
• Permission denied writing to the mount — the volume's filesystem is root-owned; set securityContext.fsGroup so the kubelet chowns it for the pod user (slow on huge volumes — fsGroupChangePolicy: OnRootMismatch).

Related tools and reading¶

• On-site: YAML validator.
• Related posts: Docker volumes vs bind mounts, Node pressure and evictions.