Failover and backups
Recovering when a database dies.
It is 2 a.m. and the primary database of the delivery app crashes. Reads can still work from replicas, but nothing can be written: no new orders, no status updates. Recovery means promoting a replica to become the new primary, and pointing everyone at it. That process is failover.
A typical failover
Step 1 of 5
Detect the failure
A monitor pings the primary every second or so. After several missed checks (say 10 seconds), it decides the primary is gone. Waiting avoids false alarms, but every second of waiting is downtime.
The data you lose depends on how replication ran. With asynchronous replication the primary confirms writes before replicas have them, so the last moments of writes may exist only on the dead primary. With synchronous replication the primary waits for a replica to confirm each write, so nothing confirmed is lost, but every write now waits for a round trip to that replica.
Sync mode tradeoff
Run this in sync mode and compare it with async at the same lag. Notice that no replica read is stale, but writes now wait. Count the total wait across the four writes, then try setting the lag to 0.
- t=0sWrite to primary
- t=1sRead from replica
- t=3sWrite to primary
- t=4sRead from replica
- t=6sWrite to primary
- t=7sRead from replica
- t=9sWrite to primary
- t=10sRead from replica
Four writes each waited 2 seconds, 8 seconds in total, in exchange for zero stale reads and no confirmed write lost. (The simulator exaggerates by using seconds; real round trips are milliseconds, but the shape of the tradeoff is the same.) Many systems compromise: one synchronous replica for safety, the rest asynchronous for speed.
Your turn
Your checkout needs no confirmed order ever to be lost, but reading menus can tolerate staleness. Describe the replication setup you would choose.
Stuck? Mix strong and relaxed settings.
What to remember
Failover promotes a replica when the primary fails.
Detection, promotion, redirection and fencing are all required.
Async replication can lose recent writes; sync waits on every write.
Rehearse failover before you need it.