Disaster Recovery: RPO, RTO, and the Four Strategies

Introduction¶

Disaster recovery planning starts with two numbers, agreed with the business, not the engineers:

• RPO (Recovery Point Objective) — the maximum acceptable data loss, measured in time. "We can lose up to 15 minutes of transactions."
• RTO (Recovery Time Objective) — the maximum acceptable downtime. "We must be back within 1 hour."

Tighter numbers cost more. Every strategy below is a point on the cost/RTO curve.

The four strategies¶

1. Backup and restore¶

Backups (and infrastructure-as-code) in a second Region. On disaster, you provision everything and restore data.

• RPO: your backup frequency (hours, unless you also ship transaction logs).
• RTO: hours — you're building the environment from scratch.
• Cost: lowest — just storage.
• Fit: internal tools, anything where a day's outage is survivable.

2. Pilot light¶

Core data is continuously replicated to the DR Region; minimal always-on infrastructure (a small DB replica, AMIs, IaC). Compute is off until needed.

• RPO: seconds–minutes (async replication).
• RTO: tens of minutes — scale up the pre-defined compute, flip DNS.
• Cost: low-moderate — you pay for the replica + storage.

3. Warm standby¶

A scaled-down but running copy of the full stack in DR. It can serve traffic immediately, then scales up.

• RPO: seconds.
• RTO: minutes — it's already up; scale and cut over.
• Cost: moderate–high — a running (if small) second environment.

4. Multi-site active/active¶

Full production in 2+ Regions, both serving traffic, data replicated bidirectionally or via a global database.

• RPO: near-zero.
• RTO: near-zero — the surviving Region already has the load (or takes it via Route 53).
• Cost: highest — 2× everything, plus the complexity of multi-master data.

Data replication choices (AWS)¶

• RDS: cross-region read replica (pilot light), or Aurora Global Database (~1s RPO, <1min promotion).
• DynamoDB: global tables (active/active, multi-Region).
• S3: Cross-Region Replication (async, minutes).
• EBS: snapshots copied cross-Region, or a replication tool.
• Whole-workload: AWS Elastic Disaster Recovery (DRS) — continuous block replication of servers, pilot-light style.

The runbook and the test¶

A DR plan you haven't executed is a hypothesis. You need:

• A written, step-by-step runbook (who declares a disaster, the exact commands / console steps, the DNS cutover, how to validate).
• Regular game days — actually fail over to DR, run real traffic, measure actual RPO/RTO, then fail back. Quarterly is a common cadence.
• Automated cutover where possible (Route 53 failover records + health checks, or a runbook in Systems Manager).

Things that break in real failovers: IAM roles/policies that only exist in the primary Region, hardcoded Region strings, secrets not replicated, ACM certs (Region-specific), quota limits in the DR Region you never raised, and DNS TTLs set to 24h.

Verification and troubleshooting¶

• Test result: RTO much longer than target — too much manual work, or a dependency you provision from scratch. Move toward pilot light / warm standby, or pre-bake more (AMIs, IaC applied and ready).
• Test result: RPO worse than expected — replication lag under load, or a data store that isn't replicated at all (a cache with authoritative data, a file on an instance). Inventory every stateful thing.
• Failed over but the app is broken — Region-specific config: KMS keys, ACM certs, IAM, Secrets Manager, SQS/SNS ARNs, hardcoded endpoints. Parameterise Region everywhere.
• Can't fail back — no reverse replication set up. Plan fail-back from the start.
• Quota errors in DR during the game day — request limit increases in the DR Region ahead of time; they're per-Region.
• DNS cutover slow — lower the TTL on failover-relevant records to 60s before you need it.

Related tools and reading¶

• On-site: DNS propagation checker, Monitors / uptime.
• Related posts: RDS vs Aurora, and how failover works, MySQL backups with Percona XtraBackup.