SproutStackSproutStack home
BeginnerKubernetes

A database blip restarts every pod

The liveness probe checks the database, so a short outage kills the whole fleet.

The setup today

An API runs as 12 pods. Its single health endpoint checks that the process is up and that it can query the database. The same endpoint is used for both probes.

Current configuration

livenessProbe:  GET /health  every 10s, failureThreshold 3
readinessProbe: GET /health  every 10s
/health: returns 500 if a test query to the database fails

What you know

  • Last week the database failed over, which took about 40 seconds.
  • Every pod restarted, and the API took four minutes to recover after the database was back.

The new requirement

A short database outage should cause errors for those seconds, not a fleet-wide restart and a long recovery.

Make the call

2 questions · first-try answers earn the most XP

1. What do you change?
2. If readiness checks the database, what happens during the next 40-second failover?
What a strong answer also covers
  • Use a startup probe for slow-starting apps, so liveness does not kill them while they boot.
  • Have the app fail fast with clear errors when a dependency is down, rather than hanging on timeouts.
  • Probes should be cheap: no heavy queries on a 10-second schedule across every pod.
  • Restart storms also hit dependencies: hundreds of pods reconnecting at once can knock over a recovering database.
Variations to think through

Change one fact and decide again. There is no single right answer here.

  • The app sometimes deadlocks and stops responding. How should liveness detect that without checking the database?
  • The API can still serve cached responses when the database is down. What should readiness say then?
  • Startup takes 90 seconds to warm caches. Which probe protects it?

The background

More drills