API infrastructure & backend engineering

English

Resilience

Designing failover around RPO and RTO, not around diagrams

Architecture diagrams with two regions and arrows between them are easy to draw. Whether the system actually survives the loss of a zone, a database or a dependency depends on decisions that the diagram does not show: how much data you can lose, how long you can be down, and whether anyone has ever practised the switch.

Published 6 min read

Start with the objectives

Resilience work often starts with technology – a second region, a managed database with replicas – and only later asks what it is for. The order should be reversed. For each business capability, agree two numbers with its owners: the RPO, the maximum period of data you can afford to lose, and the RTO, the maximum time the capability may be unavailable. ISO 22301 uses these terms in business continuity management, and they translate directly into engineering requirements.

The numbers differ widely between capabilities. A ledger of customer balances may need an RPO of zero; a product-recommendation cache may tolerate losing everything and rebuilding it. Treating every component as equally critical makes the system expensive and the failover procedure unmanageably complex.

Replication choices follow from RPO

Synchronous replication acknowledges a write only after it is durable in more than one location. It can deliver an RPO close to zero, but every write pays the round-trip time between locations, and if the replica is unreachable the primary must either block or degrade. Across zones in one region the cost is usually acceptable; across distant regions it often is not.

Asynchronous replication acknowledges writes locally and ships them afterwards. Writes are fast, but a failover loses whatever had not yet replicated – the replication lag becomes your effective RPO. Monitor that lag continuously and alert when it exceeds the agreed RPO, because an unnoticed lag of an hour silently changes your recovery promise.

Event logs and queues need the same analysis as databases. A broker that replicates asynchronously between regions has its own RPO, and consumers must be able to handle replays and gaps after a failover.

Topology follows from RTO

An RTO of several hours allows a cold standby: infrastructure defined as code, backups restored on demand. An RTO of minutes needs a warm standby with running infrastructure and continuously replicated data. An RTO near zero needs active-active operation, where both sites serve traffic all the time.

Active-active sounds like the best option, but it has strict preconditions. Each site must be able to carry the full peak load alone, or a failover merely moves the outage. Data that is written in both sites needs a conflict strategy – partitioning ownership by customer or entity, or using data types that merge safely. And global load balancing must detect unhealthy sites quickly without flapping on transient errors.

Dependencies are part of the topology

A service replicated across two zones is not resilient if both copies depend on a single authentication server, secrets store, DNS provider or third-party API. List every dependency on the critical path, including the boring ones, and decide for each: is it replicated with the same objectives, can the service degrade gracefully without it, or is it an accepted single point of failure with a documented risk?

Graceful degradation is often the most effective tool. A checkout that can accept orders and queue confirmation emails while the email provider is down keeps its RTO; one that fails the request does not.

Automate detection, decide on promotion

Detecting failure should be automated and fast: health checks that test real dependencies, not just a process that responds to pings. Promoting a replica to primary is more delicate. Automatic promotion reduces RTO but risks split-brain – two primaries accepting conflicting writes – during network partitions. Use mechanisms with fencing, where the old primary is guaranteed to stop accepting writes, or keep promotion as a deliberate, scripted human decision with a clear runbook when the RTO allows it.

Write runbooks as executable steps with expected outputs, not as prose. Every manual step is a place where a tired operator at 03:00 makes a mistake.

Rehearse, measure, repeat

Game days turn assumptions into measurements. Start in a staging environment: remove a zone, fail over the database, block a dependency, expire a certificate. Measure actual recovery time and data loss and compare them with the objectives. Then move carefully into production with limited blast radius – a single service, a quiet period, a rollback plan.

Every exercise produces findings: an alert that did not fire, a runbook step that no longer matches the infrastructure, a dependency nobody knew about. Track them like any other engineering backlog. Over time, the recovery procedure becomes routine instead of heroic.

Backups are the last line

Replication protects against infrastructure failure, not against mistakes. A faulty migration or a malicious deletion replicates instantly to every copy. Point-in-time backups, stored separately and protected against deletion with immutability or separate credentials, remain essential. Test restores regularly and measure how long they take; for large databases, restore time alone can exceed the RTO, which is a design finding in itself.

Communicating during failover

Technical recovery is only part of an incident. Customers, partners and internal teams need to know what is happening, and confused communication can extend the business impact of an outage that was handled well technically. Prepare status-page templates and internal notification lists in advance, define who declares a failover and who communicates it, and make sure those tools do not depend on the infrastructure that has failed.

After the event, a blameless post-incident review should compare the actual RPO and RTO with the objectives, document the timeline from detection to recovery, and turn every surprise into an action item with an owner.

Worked example: Ledger and catalogue with different objectives

A platform ran a transaction ledger and a product catalogue in one region. Owners agreed on an RPO of zero and an RTO of 15 minutes for the ledger, and an RPO of 24 hours and an RTO of four hours for the catalogue. The ledger moved to synchronous replication across three zones with automated, fenced failover; write latency rose by about 2 ms. The catalogue stayed with nightly backups and infrastructure as code for a cold rebuild. A quarterly game day removing one zone measured ledger recovery at six minutes with no data loss; the first exercise also revealed that the secrets store ran in a single zone, which was then replicated.

Common pitfalls

  • Building a second region before agreeing on RPO and RTO.
  • Sizing active-active sites so that each needs the other to handle peak load.
  • Ignoring replication lag, which silently becomes the real RPO.
  • Replicated services that share a single-zone dependency.
  • Runbooks that have never been executed end to end.

Questions & answers

Is multi-region always better than multi-zone?

No. Multi-zone within one region covers the most common failures with low latency cost. Multi-region adds protection against regional outages but brings latency, data-consistency and cost trade-offs that need a clear business reason.

Should failover be fully automatic?

Detection should be. Promotion of stateful components can be automatic when fencing prevents split-brain; otherwise a scripted, human-confirmed step is often the safer choice.

How often should we rehearse?

At least after every significant architecture change and on a regular schedule – quarterly is common for critical systems.

Checklist

  • Agree RPO and RTO per business capability.
  • Choose replication mode per data store from the RPO.
  • Choose cold, warm or active-active topology from the RTO.
  • Size each site for full peak load where active-active is used.
  • Inventory dependencies and define degraded modes.
  • Automate detection; fence or script promotion.
  • Rehearse failover and restore, and track findings.

Primary sources

  • ISO 22301:2019 – Security and resilience – Business continuity management systems
  • Beyer, B. et al. (eds.), Site Reliability Engineering, O’Reilly, 2016 – chapters on managing incidents and data integrity
  • Kleppmann, M., Designing Data-Intensive Applications, O’Reilly, 2017 – chapter 5, Replication
  • PostgreSQL documentation – High Availability, Load Balancing, and Replication

All research