Discord's hot partitions and trillions of messages
One busy channel slows the whole Cassandra cluster. Discord rebuilds the path in front of the database.
The setup at Discord
Discord stored messages in Cassandra, partitioned by channel and a fixed time window (bucket), so a channel's recent messages live together on a few replicas. Reads and writes used quorum consistency. The cluster grew from 12 nodes in 2017 to 177 nodes holding trillions of messages by early 2022.
How it was configured
partition key: (channel_id, bucket) # bucket = static time window
replication factor: 3, quorum reads and writes
Cassandra on the JVM, with heavy GC tuningWhat you know
- A big server announcement (@everyone) sends huge numbers of users to read the same channel at once.
- On-call was paged often, and maintenance like compaction kept falling behind.
What happened next
Make the call
2 questions · first-try answers earn the most XP
What actually happened
Discord first built data services in Rust that sit between its API and the databases. Their main feature is request coalescing: if many users request the same row at the same time, the database is queried once. Requests are routed with consistent hashing on the channel ID, so requests for one channel meet on the same instance.
Discord then moved its messages from Cassandra to ScyllaDB, a Cassandra-compatible database written in C++ without a garbage collector, after migrating its other databases first. Its own Rust migrator copied data at up to 3.2 million messages a second and finished the trillions of messages in about nine days, apart from a few stubborn ranges at the end.
The result: 177 Cassandra nodes became 72 ScyllaDB nodes. The p99 latency for fetching historical messages went from 40–125 ms to about 15 ms, and message inserts went from 5–70 ms to a steady 5 ms.
Sources
Lessons to take away
- Protect the database with a thin service in front of it: coalescing, routing and limits do more than tuning the database alone.
- Hot keys are a data model problem as much as a capacity problem; plan for the most popular entity, not the average.
- Migrations of this size need a fast, resumable copier and a plan for the last stubborn ranges.
- Measure tail latency per partition, not only cluster averages.
Variations to think through
Change one fact and decide again. There is no single right answer here.
- A single channel gets more writes than one partition can take. Coalescing does not help writes. What now?
- How would you add a short-lived cache in the data service, and what staleness is acceptable for messages?
- Your migration copier is 99.99% done but stuck on a few huge partitions. What do you do?