Disaster Recovery (DR)

Disaster recovery (DR) is the set of policies, tools, and procedures for recovering critical technology infrastructure and data after a catastrophic event. Unlike routine failure handling (circuit breakers, retries, graceful degradation), DR addresses scenarios where an entire region, data center, or data set is compromised: natural disasters destroying a data center, region-wide cloud outages, data corruption propagating through replication, ransomware encrypting all accessible storage, or catastrophic software bugs that destroy data. DR planning is not optional for any system that stores data users depend on -- the question is not whether a disaster will occur, but when, and how prepared the organization is to recover.

DR planning revolves around two fundamental metrics. Recovery Point Objective (RPO) defines the maximum acceptable data loss, measured in time. An RPO of 1 hour means the organization accepts losing up to 1 hour of data -- the DR system must have a copy of all data from at least 1 hour ago. An RPO of zero means no data loss is acceptable, requiring synchronous replication to the DR site. Recovery Time Objective (RTO) defines the maximum acceptable downtime. An RTO of 4 hours means the system must be fully operational within 4 hours of the disaster being declared. An RTO of zero means continuous availability with no perceptible downtime, requiring active-active multi-region deployment. RPO and RTO directly determine the cost of the DR solution: tighter objectives (less data loss, less downtime) require more expensive infrastructure and more complex procedures.

DR strategies are ranked by cost and recovery speed, forming a spectrum from cheapest-and-slowest to most-expensive-and-fastest. Backup and Restore is the simplest: regularly back up data to a separate region, and when disaster strikes, provision new infrastructure and restore from backups. This is the cheapest option but has the longest RTO (hours to days) and RPO depends on backup frequency (typically hours). Pilot Light maintains a minimal standby environment with core infrastructure (database replicas, DNS entries) pre-provisioned but not actively running compute. During a disaster, compute resources are scaled up and traffic is rerouted, achieving an RTO of 10-30 minutes. Warm Standby runs a scaled-down but fully functional copy of the production environment that handles a small amount of traffic or none. Failover involves scaling up and rerouting, achieving an RTO of minutes. Multi-Site Active-Active runs the full application stack in multiple regions simultaneously, with traffic routed to the nearest healthy region. There is effectively no RTO because all regions are always serving traffic, but this is the most expensive option (2x+ infrastructure cost) and the most complex to implement correctly due to data replication and consistency challenges.

DR testing is as important as DR planning. A DR plan that has never been tested is a plan that will not work when needed. Organizations must regularly test their DR procedures: tabletop exercises (walking through the DR plan step by step), automated failover drills (actually triggering failover to the DR site and back), and full-scale DR tests (operating entirely from the DR site for a sustained period). GitLab's 2017 data loss incident is a cautionary tale: their backup procedures had silently failed, and they discovered this only when they needed to restore. Regular testing ensures that backup procedures work, failover automation functions correctly, runbooks are accurate, and team members know their roles during a disaster.

DR is like home disaster insurance. RPO is how much stuff you can afford to lose: if you back up family photos to the cloud monthly (RPO = 1 month), you could lose up to a month of photos in a fire. If you back up daily (RPO = 1 day), you lose at most a day. If your photos sync to the cloud instantly (RPO = 0), you lose nothing. RTO is how quickly you need to be in a livable space again: a hotel room tonight (RTO = hours) is cheaper than owning a fully furnished second home always ready to move into (RTO = 0). The more protection you want (less data loss, faster recovery), the more you pay. Most families choose a practical balance, not maximum protection for everything.