Horizontal Pod Autoscaling

Introduction¶

The HorizontalPodAutoscaler changes a Deployment's (or StatefulSet's) replica count to keep an observed metric near a target. CPU utilisation is the classic trigger; custom and external metrics unlock the rest.

The algorithm¶

desiredReplicas = ceil( currentReplicas × (currentMetricValue / targetMetricValue) )

If pods average 180% of their CPU request and the target is 60%, that's ceil(3 × 180/60) = 9 replicas. The controller re-evaluates every ~15s (--horizontal-pod-autoscaler-sync-period).

CPU/memory targets are a percentage of the pod's requests — so you must set resources.requests.cpu. No request → HPA can't compute utilisation → it stays put and logs FailedGetResourceMetric.

A CPU HPA¶

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100        # at most double per 15s
          periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 300   # wait 5 min of lower load before scaling in
      policies:
        - type: Pod
          value: 1
          periodSeconds: 60             # remove at most 1 pod/min

The behavior block is the anti-flap knob: scale up fast, scale down slow. The default scale-down stabilization is 300s.

Custom and external metrics¶

• Pods / Object metrics — from your app via the custom metrics API (prometheus-adapter, KEDA): requests-per-second per pod, queue length, active connections.
• External metrics — from outside the cluster: SQS queue depth, Kafka lag, Pub/Sub backlog, a CloudWatch metric.

  metrics:
    - type: Pods
      pods:
        metric: { name: http_requests_per_second }
        target: { type: AverageValue, averageValue: "100" }
    - type: External
      external:
        metric: { name: sqs_queue_depth, selector: {matchLabels: {queue: jobs}} }
        target: { type: AverageValue, averageValue: "30" }

KEDA is worth knowing — it wraps HPA with dozens of scalers (including scale-to-zero, which vanilla HPA can't do) and is the standard for event-driven workloads.

HPA + VPA + Cluster Autoscaler¶

• HPA — more/fewer pods (horizontal).
• VPA — bigger/smaller pods (vertical, adjusts requests). Don't run VPA in Auto mode on the same workload as an HPA using CPU/memory — they fight. VPA in Off (recommendation) mode alongside HPA is fine.
• Cluster Autoscaler — more/fewer nodes, triggered by pods that can't schedule (HPA created pods, no room) or nodes sitting empty. HPA and CA work together: HPA adds pods → some go Pending → CA adds a node.

Verification and troubleshooting¶

kubectl get hpa api -w
kubectl describe hpa api          # events, current/target, why it's not scaling
kubectl top pods -l app=api
kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods   # is metrics-server working?

• HPA shows <unknown>/60% — metrics-server not installed/healthy, or the pods have no CPU request. kubectl top pods must work first.
• FailedGetResourceMetric / missing request for cpu — add resources.requests.cpu to the container.
• Scales up then immediately down (flapping) — bursty metric + short stabilization. Increase scaleDown.stabilizationWindowSeconds, target a smoother metric (RPS instead of CPU), or widen the target band.
• Won't scale past N despite load — hit maxReplicas, or the nodes are full and Cluster Autoscaler isn't adding nodes (quota, node group max, or CA not installed). kubectl get pods for Pending.
• Custom metric HPA does nothing — the custom/external metrics API isn't registered (kubectl get apiservices | grep metrics), the adapter can't reach Prometheus, or the metric name/labels don't match.
• Scale-down too aggressive during a traffic dip — default is already conservative (300s); if you customised behavior, raise it.
• Rolling update + HPA interaction — during a deploy, replica count churns; HPA and the Deployment controller both adjust. Usually fine; if problematic, pause HPA during deploys or use minReadySeconds.

Related tools and reading¶

• On-site: YAML ↔ JSON converter.
• Related posts: Requests vs limits and OOMKilled, Node pressure and evictions.