Kubernetes Version Upgrades Without Drama

Introduction¶

Kubernetes ships a minor release roughly every 4 months and supports the latest 3. Falling behind means a rushed multi-hop upgrade later, so treat upgrades as routine quarterly maintenance.

The skew policy (what can differ)¶

• kube-apiserver: the newest component. In an HA control plane, instances may differ by 1 minor during the upgrade.
• kubelet / kube-proxy: may be up to 3 minor versions older than the apiserver (was 2 before 1.28), never newer.
• kubectl: within 1 minor of the apiserver.
• You upgrade one minor version at a time. No 1.27 → 1.30 jump; go 27→28→29→30.

The order¶

1. Back up etcd (etcdctl snapshot save) and your manifests / Helm values.
2. Upgrade the control plane — apiserver, controller-manager, scheduler, then etcd if needed. (Managed: click the button / API call; it does HA rollout.)
3. Upgrade kubeadm and the kubelet on each node, one node at a time: drain → upgrade kubelet+kubeadm → uncordon.
4. Upgrade add-ons — CNI, CoreDNS, ingress controller, CSI drivers, metrics-server, cluster-autoscaler — to versions that support the new K8s. Check each add-on's compatibility matrix before step 2.
5. Upgrade kubectl to match.

API deprecations — the thing that actually breaks¶

Each release removes long-deprecated API versions. A manifest using a removed version fails to apply, and worse, an installed object on a removed API can block the upgrade or vanish from kubectl get.

# Find deprecated/removed APIs in use BEFORE upgrading
kubectl deprecations         # (krew plugin: kube-no-trouble / "kubent")
kubent
pluto detect-all-in-cluster

Classic breakages: networking.k8s.io/v1beta1 Ingress, PodSecurityPolicy (removed 1.25 — migrate to Pod Security Admission), batch/v1beta1 CronJob, autoscaling/v2beta* HPA, policy/v1beta1 PDB. Fix your manifests and the live objects (re-apply on the new version) first.

Safe node draining¶

kubectl drain node-3 --ignore-daemonsets --delete-emptydir-data --timeout=300s
# ... upgrade kubelet, reboot if needed ...
kubectl uncordon node-3

• drain respects PodDisruptionBudgets. Every stateful or minimum-availability workload should have one:

  apiVersion: policy/v1
  kind: PodDisruptionBudget
  metadata: { name: api }
  spec:
    minAvailable: 2          # or maxUnavailable: 1
    selector: { matchLabels: { app: api } }
  

• A drain that hangs is usually a PDB that can't be satisfied (only 2 replicas, minAvailable: 2) or a pod with no controller. Fix the PDB or the workload.
• DaemonSet pods aren't evicted (--ignore-daemonsets is required).
• For managed node groups, the cloud does surge upgrades: new node up, old node drained, repeat. Set maxSurge/maxUnavailable on the node pool.

Test first¶

• Run the upgrade on a staging cluster of the same version and add-ons.
• Or spin an ephemeral cluster (kind/k3d) at the target version and kubectl apply your manifests to catch API breakage.
• Read the release notes "Urgent Upgrade Notes" and "Deprecations" sections for every version you're passing through.

Verification and troubleshooting¶

kubectl version
kubectl get nodes                 # all Ready, all on the new version
kubectl get pods -A | grep -v Running
kubectl get apiservices | grep -v True     # broken aggregated APIs
kubectl -n kube-system get pods            # CoreDNS, CNI, kube-proxy healthy

• kubeadm upgrade plan says an add-on is incompatible — upgrade CoreDNS / kube-proxy versions it manages, or skip with --ignore-preflight-errors knowingly.
• Nodes NotReady after kubelet upgrade — CNI plugin version too old for the new kubelet, kubelet config schema change, or cgroup driver mismatch (systemd vs cgroupfs — must match the container runtime).
• Workloads fail to schedule post-upgrade — a removed API (PSP → PSA now enforcing), a new default (e.g. seccomp), or a NodeAffinity referencing a label the new nodes lack.
• kubectl get ingress empty after upgrade — Ingress objects were on v1beta1; migrate to networking.k8s.io/v1 and re-apply.
• etcd upgrade issues — always have the snapshot; managed clusters handle this, kubeadm upgrades etcd as a static pod.
• Drain stuck forever — PDB unsatisfiable, or terminationGracePeriodSeconds huge, or a finalizer. kubectl get pdb -A; --force --grace-period=0 only as a last resort.

Related tools and reading¶

• On-site: YAML ↔ JSON converter.
• Related posts: The Kubernetes pod lifecycle, kubectl commands you use daily.