Disaster Recovery and Backups: RPO, RTO, Multi-Region Failover, and DR Testing
Learn how to design for disaster recovery: RPO vs RTO, full/incremental/differential backup strategies, multi-region failover architecture, active-active vs active-passive, DR tiers, and game-day testing.
Why Disaster Recovery Matters
Reliability patterns like circuit breakers and retries protect you from a single request failing. Disaster recovery (DR) protects you from something much bigger: a whole data center burning down, a region-wide cloud outage, a botched migration that corrupts your primary database, or ransomware that encrypts your production data. DR is not about individual requests — it's about whether the business survives when the worst happens.
Every DR conversation starts with two numbers, and every architecture decision downstream follows from them.
Key insight: DR is a business decision disguised as an engineering problem. The right answer isn't "the most resilient architecture possible" — it's "the cheapest architecture that meets the RPO/RTO the business actually needs." Over-engineering DR wastes money; under-engineering it risks the company.
RPO and RTO
Recovery Point Objective (RPO)
RPO answers: "How much data can we afford to lose?" It's measured backward in time from the moment of failure.
Recovery Time Objective (RTO)
RTO answers: "How long can we be down?" It's measured forward from the moment of failure until service is restored.
Worked Example
Say you set RPO = 1 hour and RTO = 4 hours for an e-commerce order database:
- RPO of 1 hour means your backup/replication cadence must capture data at least every 60 minutes. If the primary dies at 2:47 PM and your last snapshot was 2:00 PM, you lose up to 47 minutes of orders. To hit this RPO you need continuous replication or backups at least hourly — a nightly backup (23-hour RPO) would violate it badly.
- RTO of 4 hours means that from the moment of failure, you have 4 hours to detect it, fail over, and serve traffic again. That drives you toward automation: a manual "page an engineer, they SSH in, restore from S3, update DNS" process rarely finishes in 4 hours under stress. You likely need a pre-provisioned standby and scripted or automatic failover.
These two numbers are independent and drive completely different investments:
| Objective | Drives investment in | Cheap way to improve it | Expensive way to improve it |
|---|---|---|---|
| RPO | Backup/replication frequency | More frequent snapshots | Synchronous multi-region replication |
| RTO | Failover automation & standby capacity | Runbooks + practice | Hot standby, automated traffic-manager failover |
RPO near zero and RTO near zero together are the most expensive combination you can ask for — it typically requires synchronous cross-region replication plus a fully warm or active standby, which multiplies infrastructure cost and adds write-latency overhead in the normal case, not just during a disaster.
Backup Strategies
Full, Incremental, and Differential
- Full backup — copies everything, every time. Simplest to restore (one file), most expensive to store and slowest to run.
- Incremental backup — copies only what changed since the last backup of any kind. Cheapest and fastest to create, but restoring means replaying the full backup plus every incremental since — a longer, more fragile restore chain.
- Differential backup — copies everything changed since the last full backup. Storage and restore time grow between fulls, but restoring only ever needs two files: the last full plus the latest differential.
Storage Cost vs Restore-Time Tradeoff
| Strategy | Backup storage cost | Backup speed | Restore complexity | Restore speed |
|---|---|---|---|---|
| Full | High (redundant data every run) | Slow | Trivial — one file | Fast |
| Incremental | Low | Fast | Complex — full + every increment in order | Slow (chain replay) |
| Differential | Medium, grows until next full | Medium | Simple — full + one differential | Medium |
A broken link in an incremental chain breaks every restore after it. If Tuesday's incremental is corrupted, Wednesday and Thursday's incrementals are useless even if they're intact, because each depends on the one before it. Most production systems use a hybrid: weekly full + daily incremental, with periodic chain-integrity checks.
Backup Schedule Example
Sunday 00:00 -> FULL backup (baseline)
Mon-Sat 00:00 -> INCREMENTAL backup (changes since previous day)
Every hour -> Transaction log backup (for point-in-time recovery)
Retention: 4 weekly fulls, 30 days of incrementals, 7 days of transaction logs
Transaction log (or write-ahead log) shipping is what lets databases hit sub-hour, even sub-minute, RPOs without doing a full backup every few minutes — you replay logs on top of the last full/incremental to reconstruct state at any point in time.
Backup Testing: Untested Backups Are Not Backups
A backup you have never restored is a hypothesis, not a safety net. Backup jobs silently fail: permissions change, storage fills up, encryption keys rotate and nobody updates the restore script, schemas drift so an old backup no longer imports cleanly.
The single most common DR failure isn't "no backups" — it's "backups that turned out not to restore." Teams discover this during the actual outage, which is the worst possible time to find out.
Restore Drills
| Practice | Why it matters |
|---|---|
| Scheduled restore drills (e.g., monthly) | Verifies the backup is actually restorable, not just "created" |
| Restore to an isolated environment | Confirms the process works without risking production |
| Time the restore | Tells you if you can actually meet your RTO |
| Checksum/row-count validation post-restore | Confirms data integrity, not just file existence |
| Rotate who runs the drill | Confirms the runbook is usable by someone other than its author |
A good rule of thumb: if you can't point to a dated log entry of the last successful restore test, treat your RTO/RPO numbers as unverified.
Backup Immutability and Ransomware
Traditional DR planning assumes the disaster is a hardware failure or a data center outage. Ransomware changes the threat model: an attacker who compromises production credentials can often reach — and encrypt or delete — your backups too, if they're just another writable location the same credentials can touch.
| Protection | What it prevents |
|---|---|
| Immutable / WORM (write-once-read-many) backup storage | Backups can't be altered or deleted even by a compromised admin account, for a set retention window |
| Separate credentials/account for backup storage | A breach of production credentials doesn't automatically grant access to delete backups |
| Air-gapped or offline copy | A network-based attack physically cannot reach a copy that isn't network-reachable |
| Versioned backups (not overwrite-in-place) | An attacker who "successfully" encrypts today's backup hasn't destroyed yesterday's |
If the same credentials that write your application data can also delete your backups, you don't have a ransomware recovery plan — you have a shared blast radius. Backup storage should be a permissions boundary, not just a different bucket.
Multi-Region Failover Architecture
How Failover Actually Happens
- Replication streams data continuously (or on a schedule) from primary to standby region — this is what determines your realized RPO.
- Health checks on the traffic manager (Route 53, Cloud DNS, a global load balancer) continuously probe the primary region.
- Detection — health checks fail past a threshold (avoiding flapping on a single blip).
- Failover — DNS/traffic manager shifts traffic to the standby region. TTLs matter here: a 1-hour DNS TTL can silently add an hour to your RTO regardless of how fast the standby is ready.
- Promotion — the standby database is promoted from replica to writable primary. This is often the riskiest scripted step, and it's why it needs to be rehearsed, not improvised.
- Fail-back — once the original region recovers, traffic is not automatically shifted back; re-syncing and cutting back over is a deliberate, separate operation to avoid a second outage.
Replication lag is a hidden RPO tax. If your replication is asynchronous and lagging 90 seconds behind at the moment of failure, your realized RPO is 90 seconds even if your target was "near zero." Monitor replication lag as a first-class metric, not an afterthought.
Active-Active vs Active-Passive
| Dimension | Active-Passive | Active-Active |
|---|---|---|
| Normal operation | Standby region idle or read-only | Both regions serve live traffic |
| Cost | Lower — standby capacity is minimally provisioned | Higher — full capacity duplicated and always running |
| Complexity | Lower — one write path | Higher — writes can land in either region |
| RTO achieved | Minutes to tens of minutes (promotion + DNS cutover) | Seconds to near-zero (traffic already flowing both places) |
| Conflict resolution burden | None — single writer | Significant — concurrent writes to the same record in two regions must be reconciled (last-write-wins, vector clocks, CRDTs, or application-level merge logic) |
| Best fit | Most systems where a few minutes of downtime is acceptable | Systems with a hard near-zero RTO requirement and the engineering budget to handle write conflicts correctly |
Active-active is frequently over-purchased. Teams reach for it because "active-active sounds more resilient," then spend months fighting conflict-resolution bugs for an RTO improvement the business never actually required. Confirm the RTO requirement first — often active-passive with a well-rehearsed failover comfortably meets it at a fraction of the complexity.
DR Strategy Tiers
A widely used way to frame the cost/speed tradeoff (popularized by AWS's DR guidance, but the shape applies to any cloud):
| Tier | Description | Typical RTO | Typical RPO | Relative cost |
|---|---|---|---|---|
| Backup & restore | Backups stored offsite/cross-region; infrastructure rebuilt from scratch on disaster | Hours to a day+ | Hours (since last backup) | $ — cheapest |
| Pilot light | Core data replicated continuously; minimal/no compute running in the standby region until needed | Tens of minutes to a couple hours | Minutes | $$ |
| Warm standby | Scaled-down but fully functional copy of the stack always running in the standby region; scale up on failover | Minutes | Seconds to minutes | $$$ |
| Multi-site active-active | Full-scale stack live in two or more regions simultaneously, serving real traffic at all times | Near zero | Near zero | $$$$ — most expensive |
Moving down this table buys you a faster, more data-current recovery — at a real dollar cost that scales roughly with how "always-on" the standby capacity is. Picking a tier is where the RPO/RTO numbers from earlier in this article turn into an actual budget line.
Testing DR With Game Days
Chaos engineering (covered separately) tests request-level resilience — killing an instance, injecting latency into one service call. DR game days test something bigger: can the organization actually survive losing an entire region?
What a DR Game Day Looks Like
| Game day type | What it validates |
|---|---|
| Tabletop exercise | Do people know the plan? Cheap, but doesn't catch technical surprises |
| Simulated region failure (non-prod) | Does the failover automation actually work end to end? |
| Full production failover drill | Does it work under real traffic and real data volume — the only test that fully validates RTO |
| Unannounced drill | Tests whether the on-call process itself works, not just the scripted steps |
Run game days on a schedule, not "when we get around to it." Infrastructure drifts — a new service gets added that isn't replicated, an IAM permission needed for failover expires, a runbook references a tool that's since been decommissioned. Quarterly is a common cadence for full failover drills; more often for the systems with the tightest RTO commitments.
Disaster Recovery Checklist
| Area | Checklist Item |
|---|---|
| Objectives | RPO and RTO defined and signed off per system, not assumed |
| Backups | Backup strategy (full/incremental/differential) matches the required RPO |
| Backups | Backups stored in a different region/account than production |
| Testing | Restore drills run on a schedule, with timing recorded |
| Replication | Replication lag monitored and alerted on |
| Failover | Failover process automated or fully scripted, not manual/ad hoc |
| Failover | DNS/traffic-manager TTLs tuned to not silently inflate RTO |
| Architecture | DR tier chosen deliberately based on RTO/RPO, not by default |
| Testing | Game days scheduled, including full failover drills, not just tabletop |
| Fail-back | Fail-back procedure defined and rehearsed, not improvised after recovery |
What to Remember for Interviews
- RPO and RTO are different axes: RPO is about data loss tolerance, RTO is about downtime tolerance — they drive different investments and should each be stated as explicit numbers, not vibes.
- Untested backups are not backups: Always mention restore drills, not just backup creation, when asked about DR.
- DR tiers trade cost for speed: Backup-and-restore → pilot light → warm standby → multi-site active-active, each step buying a faster RTO/RPO for more money.
- Active-active isn't automatically better: It buys near-zero RTO at the cost of conflict resolution and doubled infrastructure spend — justify it against the actual requirement.
- DR game days are region-level, not request-level: Distinguish this from chaos engineering, which tests individual service failures, not "we lost an entire region."
Practice: For any system design, ask "What's the RPO and RTO the business actually needs here?" before proposing an architecture. Backing into the numbers from a cool architecture is a common interview mistake — start from the requirement.