IntermediateAPI gateway & load balancing
Retries at every layer turn a slowdown into an outage
The client, gateway and service each retry three times. One slow database does the rest.
The setup today
A mobile app calls an API gateway, which calls an orders service, which queries a database. Each layer was configured independently, and each retries failed calls.
Current configuration
mobile app: 3 attempts, 1 s apart
gateway: 3 attempts per request, no backoff
orders service: 3 attempts per database queryWhat you know
- When the database slowed down for two minutes, its query rate went up, not down.
- The service took 20 minutes to recover after the database was healthy again.
The new requirement
A short slowdown in a dependency must not snowball into a long outage.
Make the call
2 questions · first-try answers earn the most XP
What a strong answer also covers
- Only retry idempotent operations, or carry an idempotency key, so retries cannot double-charge.
- Propagate deadlines: if the caller has 2 seconds left, a deep retry that needs 5 is wasted work.
- Load shedding at the service (fast 503s) protects the database better than queuing.
- Watch retry rate as a metric; a jump in retries is an early warning.
Variations to think through
Change one fact and decide again. There is no single right answer here.
- The app is old and cannot be updated for months. Which layer do you change first?
- Some calls are POSTs that create orders. How do retries stay safe there?
- Two other services call the same database. How do you protect it from all of them?