Cache Stampede (Thundering Herd)

A cache stampede, also known as the thundering herd problem, is one of the most dangerous failure modes in caching systems. It occurs when a popular cache entry expires and the flood of requests that were previously served by the cache simultaneously fall through to the database. If a key serves 10,000 requests per second and its TTL expires, all 10,000 requests in the next second will query the database with the same query, potentially overwhelming it with identical work. The database, designed to handle perhaps 1,000 queries per second total, is suddenly hit with 10x its capacity for a single key, causing cascading latency increases, timeouts, and potentially a full database outage.

The stampede problem is exacerbated by several factors. First, the most popular cache entries are the ones most likely to cause stampedes -- they serve the most traffic, so their expiry creates the largest surge. Second, the database query to regenerate the cache entry is often the most expensive query (that is why it was cached in the first place), so the database is hit with thousands of copies of its most expensive query simultaneously. Third, the stampede can cascade: the slow database responses cause the application to hold connections longer, exhausting connection pools, which causes timeouts in unrelated queries, which causes other cache entries to fail to refresh, which triggers more stampedes.

Four primary solutions address cache stampedes. (1) Mutex/lock: when a cache miss occurs, the first request acquires a lock (using Redis SETNX or similar) and refreshes the cache. Subsequent requests see the lock and either wait or return a stale value. This limits database load to one query per key per refresh cycle. (2) Probabilistic early expiration (XFetch): each request randomly decides whether to refresh the cache before TTL expires, with the probability increasing as TTL approaches. Statistically, one request will refresh early, preventing a synchronized miss. (3) Stale-while-revalidate: the cache serves stale data immediately while triggering a background refresh, eliminating the stampede entirely because no request ever experiences a true miss. (4) External refresh: a background cron job or scheduled task refreshes popular cache entries before they expire, removing the application from the refresh path entirely.

The single-flight pattern (Go's singleflight package, Node.js promise caching) is a request-level deduplication mechanism that complements these solutions. When multiple concurrent requests need the same data, single-flight ensures only one request actually executes the database query, and all other requests wait for and share the result. This is different from a distributed lock because it operates within a single process -- it deduplicates concurrent requests on the same server instance. Combined with a distributed mutex for cross-server deduplication, single-flight provides comprehensive stampede protection at both the local and distributed levels. The quantitative impact is significant: without protection, a popular key at 10,000 RPS generates 10,000 database queries in the first 100ms after expiry. With single-flight + mutex, it generates exactly 1.

A coffee shop brews a fresh pot every hour. At exactly 9:00 AM, the pot runs empty. The next 50 customers all arrive within minutes and each one asks the barista to brew a new pot. Without stampede protection, the barista tries to brew 50 pots simultaneously and the kitchen melts down. With a mutex: the first customer triggers a new brew, and a sign goes up saying 'Fresh pot brewing, please wait 5 minutes.' With stale-while-revalidate: the barista serves slightly old coffee from a thermos while brewing a fresh pot in the background -- nobody waits. With XFetch: one random customer at 8:55 AM asks for the pot to be refreshed early, so it is ready by 9:00 AM.