IntermediateKafka
Orders must survive a broker failure
Replication factor 2, acks=1, and a requirement for zero acknowledged-message loss.
The setup today
The orders topic was created early on, when the cluster had two brokers. The cluster now has three. Producers were tuned for latency a while ago.
Current configuration
topic: orders replication.factor: 2 min.insync.replicas: 1
producer: acks=1
unclean.leader.election.enable=false # defaultWhat you know
- Last quarter a broker died and about 40 orders that customers saw as confirmed were never processed.
- Producers can tolerate a few extra milliseconds per write.
The new requirement
The business requirement is now explicit: losing any single broker must never lose an order that a producer was told was written, and orders should keep flowing while one broker is down.
Make the call
2 questions · first-try answers earn the most XP
What a strong answer also covers
- Changing the replication factor of an existing topic needs a partition reassignment, so plan and throttle it.
- Producers must treat a rejected write as a failure and retry or surface it, never as success.
- Spread replicas across racks or availability zones, or one zone outage can take two copies at once.
- Alert on under-replicated partitions: one more failure there and writes stop.