Chaos Engineering Basics

Introduction¶

Chaos engineering is the practice of deliberately injecting failure into a system to find weaknesses before they cause an incident. It's not "break things randomly" — it's a controlled experiment with a hypothesis, a limited blast radius, and a rollback.

The method¶

1. Define steady state — a measurable "the system is healthy" signal: checkout success rate > 99.5%, p99 latency < 300ms, order throughput within normal range. Not "CPU is fine" — a user-facing metric.
2. Hypothesise — "if we kill one API pod, steady state is unaffected (traffic reroutes, latency blips < 5s)."
3. Inject the smallest realistic failure in the smallest scope (one pod, one AZ, 1% of traffic, staging first).
4. Observe — did steady state hold? Automatically abort if it degrades past a threshold.
5. Learn and fix — a failed hypothesis is the point. Fix the weakness, then re-run to confirm.

Blast radius control¶

• Start in staging / a canary environment.
• In prod: one instance, one AZ, a small % of traffic, during business hours (so people are watching), with an abort condition wired to your steady- state metric.
• Have a big red button — a one-command halt/rollback for the experiment.
• Announce it. Chaos experiments that surprise on-call erode trust.

Starter experiments¶

Experiment	Verifies
Kill a random pod	pod restarts cleanly, traffic reroutes, no dropped requests (readiness + preStop)
Kill a node / cordon+drain	pods reschedule, PDBs respected, capacity headroom exists
Add 200ms latency to the DB	timeouts/retries are sane, connection pool copes, no cascading failure
Drop 5% of packets to a dependency	retries + circuit breaker work; the dependency's failure is contained
Make a dependency return 500s	graceful degradation, not a full outage; error budgets
Fill a disk / exhaust a connection pool	alerts fire, the failure is isolated
Fail over the primary database	RTO/RPO match expectations, app reconnects (see RDS failover)
Expire a certificate in staging	monitoring catches it before users do

Tools¶

• Kubernetes: Chaos Mesh, LitmusChaos, kube-monkey. Inject pod-kill, network latency/loss/partition, IO faults, stress, DNS chaos — as CRDs.
• Cloud/infra: AWS Fault Injection Service (FIS), Gremlin (commercial), Chaos Toolkit (vendor-neutral, hypothesis-driven).
• Ad hoc: tc qdisc for latency/loss, stress-ng for CPU/memory, iptables -j DROP for partitions, kill -STOP to freeze a process.

Game days¶

A scheduled, hands-on session where the team runs experiments together (or tabletop-simulates a scenario) and practises the response:

• Pick a scenario ("region X goes dark", "the payment provider is down").
• Run it (or walk through it) with the actual runbooks.
• Time the detection and recovery; note every "we didn't know where to look" and "the runbook was wrong".
• Fix those. Re-run next quarter.

Game days are where you find that the DR runbook references a decommissioned tool, or that only one person knows the failover procedure.

Turning findings into fixes¶

Every failed hypothesis produces work:

• Missing retry/timeout/circuit breaker → add it.
• No graceful degradation → add a fallback (cached data, a "reduced functionality" mode).
• Cascading failure → bulkheads (separate pools/quotas per dependency).
• Slow detection → a better alert (symptom-based, see SLO burn-rate alerts).
• Manual, slow recovery → automate it.
• Insufficient capacity headroom → adjust autoscaling / reservations.

Track them like bugs. A chaos program that finds problems and doesn't fix them is theatre.

Verification and troubleshooting¶

• Experiment caused a real outage — blast radius was too big, no abort condition, or run in prod before staging. Scope down, wire the automatic halt, do a post-incident review like any outage.
• Steady state "held" but you're not sure — the metric wasn't sensitive enough, or the experiment was too weak (killed a pod that had no traffic). Make the steady-state SLI reflect real user experience; make the fault realistic.
• Can't get buy-in — start with staging and read-only "observability" chaos (does the alert fire?), show value, then propose small prod experiments with leadership present.
• Chaos tool itself is flaky — validate the injection actually happened (the added latency shows in traces; the killed pod shows in events) before trusting the result.
• On-call burned out by chaos — you're running too often, or not announcing, or not fixing findings so the same failure recurs. Cadence + follow-through.

Related tools and reading¶

• On-site: Monitors / uptime.
• Related posts: Disaster recovery: RPO, RTO, and the four strategies, SLO burn-rate alerts.