Facebook disappears from the internet
A maintenance command takes down the backbone, and DNS withdraws itself by design.
The setup at Facebook (Meta)
Facebook's data centers are linked by a global backbone network. At the edge, its DNS servers announce their addresses to the internet with BGP. As a health check, a DNS server stops announcing (withdraws its BGP routes) if it cannot reach the data centers, so users are sent to healthier locations.
How it was configured
backbone: links all data centers
edge DNS: announces routes via BGP; withdraws them if data centers are unreachable
internal tools: rely on the same network and DNSWhat you know
- Commands that change the backbone pass through an audit tool meant to block dangerous ones.
- Data centers are built with strict physical security.
What happened next
Make the call
2 questions · first-try answers earn the most XP
What actually happened
With the backbone down, Facebook's data centers were disconnected from each other and from the internet. The DNS servers then disabled their BGP advertisements, as designed, which made them unreachable even though they were still running. The rest of the internet could no longer find Facebook, Instagram or WhatsApp.
The outage also broke the internal tools engineers would normally use. Both primary and out-of-band network access were down, so engineers were sent onsite, and the strict physical and system security of the facilities slowed them down.
Once the backbone was back, services had to be restored carefully: data centers were reporting power-usage dips of tens of megawatts, and reversing that all at once could stress electrical systems and caches. Facebook credited its regular "storm" drills, which simulate large failures, with helping the team bring things back safely.
Sources
Lessons to take away
- A health check that every instance can fail at the same moment can turn a partial failure into a total one.
- Audit tools are safety systems; test them with the dangerous commands they exist to block.
- Out-of-band access must not share failure modes with the production network.
- Bringing a huge system back at once is its own risk; staged restoration needs practice too.
Variations to think through
Change one fact and decide again. There is no single right answer here.
- How would you change the DNS health rule so a global backbone failure does not withdraw every server?
- Your company has three data centers and a VPN for remote access. Where are your shared failure points?
- What would a drill that simulates losing the backbone look like for your team?