Systems offlineAdvanced
The failover that lost orders
Last night the primary database crashed and the standby took over. This morning: confirmed orders are missing, some order IDs exist twice, and a third of the app servers are erroring. Six parts of the setup have a weakness. Pick any, read its evidence, and choose the fix.
Online 0 / 6Down 0Not checked yet 6XP earned 0
Pick a component
Each component has one question, so read it carefully. Right answers bring it back online.
What users can do
- DownConfirmed orders survive a crashNeeds Primary, Standby
- DownExactly one primary after failoverNeeds Failover manager, Cluster vote
- DownApps write to the new primaryNeeds App servers
- DownA bad delete can be undoneNeeds Backups
What this incident covers
- App servers (connection pools): DNS, HTTP, TCP in 10 Minutes
- Primary (db-a): Replication & Sharding
- Standby (db-b): Replication & Sharding
- Failover manager (promotes a standby): Distributed Consensus: Paxos & Raft
- Cluster vote (decides who leads): CAP & Consistency in Plain Words
- Backups (recovery): Multi-Region & Disaster Recovery