Blue-Green vs Canary Deployments

Introduction¶

Both blue-green and canary aim to release new code without downtime and with a fast way back. They differ in how traffic moves and what you're watching while it does.

Rolling update (the baseline)¶

Kubernetes' default: replace pods a few at a time (maxUnavailable, maxSurge). New and old versions serve simultaneously during the roll. Simple, no extra infra. Weaknesses: rollback is another slow roll, and you can't easily gate on metrics mid-rollout.

For many services this is enough. Reach for blue-green or canary when the blast radius of a bad deploy is high.

Blue-green¶

Two identical environments. "Blue" serves production; you deploy the new version to "green", smoke-test it out of band, then flip the router so 100% of traffic hits green. Blue stays warm as an instant rollback.

Mechanics:

• LB / DNS: swap the target group (AWS), the upstream (Nginx), or the weighted DNS record. DNS is slow to propagate — prefer LB-level.
• Kubernetes: two Deployments (app-blue, app-green); the Service selector has a version: label you patch to flip.

kubectl patch service app -p '{"spec":{"selector":{"version":"green"}}}'

Pros: instant cutover, instant rollback, test green fully before traffic. Cons: 2× resources during the window; a stateful backend shared by both must tolerate both code versions at once (see migrations below).

Canary¶

Route a small slice of traffic (1%, 5%, 25%…) to the new version, watch error rate / latency / business metrics, and promote or abort.

Mechanics:

• Service mesh (Istio, Linkerd) or an ingress that supports weighting (nginx canary-weight annotation, Traefik, Gateway API) splits traffic by percentage or header.
• Progressive delivery controllers (Argo Rollouts, Flagger) automate the ramp: define steps and metric checks, the controller advances or rolls back.

# Argo Rollouts (abridged)
strategy:
  canary:
    steps:
      - setWeight: 5
      - pause: { duration: 10m }
      - analysis: { templates: [{ templateName: error-rate }] }
      - setWeight: 25
      - pause: { duration: 10m }
      - setWeight: 100

Pros: real production traffic, limited blast radius, automatic metric gating. Cons: more infra; needs good metrics; slow full rollout; sticky sessions and caching complicate "5% of users".

The database problem (both strategies)¶

While two code versions run against one database, every schema change must be backward compatible for at least one release. The expand/contract pattern:

1. Expand: add the new column/table, nullable, no drop. Deploy code that writes both old and new, reads old.
2. Deploy code that reads new.
3. Contract: a later release removes the old column once nothing uses it.

Never combine "drop column" with the same deploy that stops using it — the old pods still running will error.

Choosing¶

Verification and troubleshooting¶

kubectl argo rollouts get rollout app --watch
kubectl get svc app -o jsonpath='{.spec.selector}'
# LB: check target group health / weights in the cloud console

• Blue-green flip but some users still on old — DNS TTL caching, or long-lived connections (websockets, gRPC) that don't reconnect. Drain them or accept a tail.
• Canary metrics look fine at 5% but break at 100% — resource limits, connection-pool or DB-connection ceilings only hit at full load, or a cache cold-start. Add a load-proportional step and watch saturation metrics.
• Rollback didn't help — the bad migration already ran. This is why expand/contract matters.
• Canary gets no traffic — the weight annotation/mesh rule isn't matching, or the canary Service has no ready endpoints.
• Costs doubled and stayed there — blue environment never scaled down after a successful cutover. Automate the teardown.

Related tools and reading¶

• On-site: HTTP Header Analyzer, Uptime / status check.
• Related posts: Trunk-based development, Safe online schema changes.