SeniorAPI gateway & load balancing
A deep health check takes the whole fleet out
Every target checks the cache cluster; when the cache blips, the load balancer reacts badly.
The setup today
An application load balancer spreads traffic over 20 instances. Each instance's health check verifies the process, the database connection and the shared Redis cache cluster. Deploys deregister instances and stop them.
Current configuration
health check: GET /health every 10 s, unhealthy after 2 failures
/health: process OK + SELECT 1 + Redis PING
deregistration delay: 0 s (changed to speed up deploys)What you know
- When Redis failed over, every instance failed its health check within 20 seconds.
- Deploys cause a burst of reset connections.
The new requirement
A shared dependency blip must not change how traffic is spread, and deploys must not cut off requests in progress.
Make the call
2 questions · first-try answers earn the most XP
What a strong answer also covers
- Expose dependency health separately (for dashboards and alerts), and keep the load balancer's check shallow.
- Let instances degrade gracefully when Redis is down: serve from the database with lower limits, or return fast errors.
- The deregistration delay should cover the longest normal request, not the default by habit.
- Test it: kill a dependency in staging and watch what the load balancer does.
Variations to think through
Change one fact and decide again. There is no single right answer here.
- One instance has a bad disk and serves errors, but its process is up. How does a shallow check catch it?
- Requests can take five minutes (large uploads). What deregistration delay would you choose?
- The fleet runs behind two load balancers in different regions. What should regional failover depend on?