SproutStackSproutStack home
IntermediateCaching

The database spikes every minute

A hot homepage key expires every 60 seconds and thousands of requests rebuild it at once.

The setup today

The homepage shows "today's deals", built by a database query that takes about 2 seconds. The result is cached in Redis for 60 seconds. Category pages use the same pattern, one key per category.

Current configuration

GET homepage:deals  ->  miss  ->  run 2-second query  ->  SET homepage:deals ... EX 60
category:{id}:deals  ->  same pattern, 500 categories
# all category keys are written at deploy time with EX 600

What you know

  • The homepage gets 20,000 requests a second.
  • Every minute, database CPU jumps to 100% for a few seconds and p99 latency spikes.
  • Ten minutes after each deploy there is an even larger spike.

The new requirement

A sale will triple traffic next week. The database must not see these spikes, and the homepage must stay fast.

Make the call

2 questions · first-try answers earn the most XP

1. Why does the database spike every minute, and what fixes it?
2. What causes the bigger spike ten minutes after each deploy?
What a strong answer also covers
  • Serving stale-while-revalidate (return the old value, refresh in the background) keeps latency flat during rebuilds.
  • Give the rebuild lock a timeout so a crashed rebuilder cannot block refreshes forever.
  • For a few very hot keys, refresh them on a schedule (refresh-ahead) and never let them expire.
  • Facebook's memcache paper describes leases, which handle both stampedes and stale sets.

The background

More drills