A small capacity add breaks AWS Kinesis
Adding servers pushes every server past an operating system thread limit.
The setup at Amazon Web Services
Kinesis Data Streams has a front-end fleet of many thousands of servers that route requests to back-end clusters. Each front-end server keeps a cache of which back-end owns which shard (the shard-map), and keeps an operating system thread for each other front-end server it talks to.
How it was configured
front-end fleet: many thousands of servers
per server: one OS thread for every other front-end server
shard-map cache: rebuilt on startup; new servers are learned over up to an hourWhat you know
- Several other AWS services, including Cognito and CloudWatch, send data through Kinesis.
What happened next
Make the call
2 questions · first-try answers earn the most XP
What actually happened
The new capacity caused every front-end server to exceed the maximum number of threads allowed by an operating system setting. Shard-map construction failed, and servers were left unable to route requests to the back-end clusters. The variety of errors slowed diagnosis.
Recovery required restarting the fleet, but rebuilding shard-maps competes with serving requests, so servers had to come back slowly. The first group took traffic at 10:07 AM PST, and Kinesis returned fully to normal at 10:23 PM PST.
Services built on Kinesis were hit too. Cognito buffered its Kinesis data as a best-effort path, but the long outage filled those buffers and blocked its web servers, causing errors for sign-ins. CloudWatch, which processes metrics and logs through Kinesis, was also affected.
AWS's follow-ups included moving to larger servers (fewer servers, so fewer threads each), fine-grained alarms on thread use, testing a higher thread limit, moving the cache to a dedicated fleet, and speeding up the cellularization of the front-end fleet.
Sources
Lessons to take away
- Anything whose cost per server grows with the size of the fleet has a cliff; find it before growth does.
- Alarm on the resources that limits apply to (threads, file handles, connections), not just CPU and memory.
- Best-effort dependencies must stay best-effort: buffers that block on a downstream service turn its outage into yours.
- Recovery can be slow when warming caches competes with live traffic; plan how to restart in stages.
Variations to think through
Change one fact and decide again. There is no single right answer here.
- Your service keeps a connection from every instance to every other instance. At what fleet size does it hurt?
- A logging dependency is down for an hour. How do you make sure your service keeps working?
- How would you test a new limit (threads, connections) safely before relying on it?