Cache Stampede Prevention: Protecting Your Cache from Thundering Herds

A cache stampede happens when a popular cache entry expires and dozens (or hundreds) of requests all miss simultaneously, each one hitting your database to rebuild the same value. Your cache was supposed to protect the database — but instead, the expiration creates a thundering herd that overwhelms whatever you were trying to shield.

This happens more often than you’d think. A single cache key for “top 100 products” or “user session data” can expire at a bad moment and send a flood of requests to your database. The fix isn’t just “use a longer TTL.” You need actual stampede prevention patterns.

This guide covers the main approaches: single-flight, request coalescing, probabilistic early expiration, and lock-based cache refresh.

Why Cache Stamps Happen

The classic scenario: you cache a result with a 60-second TTL. At second 60, 200 requests arrive within milliseconds of each other. Every single one checks the cache, finds nothing, and rushes to the database.

# This code stampedes
def get_product_catalog():
    cached = redis.get("product_catalog")
    if cached:
        return json.loads(cached)

    # 200 requests arrive here at the same time
    result = db.query("SELECT * FROM products")
    redis.setex("product_catalog", 60, json.dumps(result))
    return result

The cache didn’t help at all. Every request still hit the database.

Single-Flight Pattern

The single-flight pattern ensures only one request fetches the data while all others wait for that same result. Concurrent requests to the same key share a single in-flight request.

Instead of 200 database calls, you get 1. The other 199 requests wait on the same task.

import asyncio
import httpx
from collections import defaultdict
from datetime import datetime

class SingleFlight:
    def __init__(self):
        self.in_flight: dict[str, asyncio.Task] = {}
        self.lock = asyncio.Lock()

    async def get(self, key: str, fetch_fn) -> any:
        # Check if request is already in flight
        async with self.lock:
            if key in self.in_flight:
                # Another request is already fetching this key
                task = self.in_flight[key]
            else:
                # No in-flight request, create one
                task = asyncio.create_task(self._fetch(key, fetch_fn))
                self.in_flight[key] = task
                task.add_done_callback(
                    lambda _: self._cleanup(key)
                )

        return await task

    async def _fetch(self, key: str, fetch_fn) -> any:
        result = await fetch_fn()
        return result

    def _cleanup(self, key: str):
        self.in_flight.pop(key, None)

Go Single-Flight Implementation

Go’s golang.org/x/sync/singleflight is the canonical implementation:

var sf singleflight.Group

func getProductCatalog() ([]Product, error) {
    v, err, _ := sf.Do("product_catalog", func() (interface{}, error) {
        // Only one request reaches the database
        return fetchFromDB()
    })
    return v.([]Product), err
}

Request Coalescing

Request coalescing takes single-flight further by batching multiple requests that arrive within a window. Instead of one request per key, you wait a short time (e.g., 5ms) to collect all requests for the same key, then make a single database call.

The window_ms tuning matters. Too short and you don’t coalesce enough. Too long and latency suffers.

import asyncio
from collections import defaultdict
import time

class RequestCoalescer:
    def __init__(self, window_ms: float = 5.0):
        self.window_ms = window_ms
        self.pending: dict[str, asyncio.Queue] = {}
        self.lock = asyncio.Lock()

    async def get(self, key: str, fetch_fn) -> any:
        queue: asyncio.Queue = None

        async with self.lock:
            if key not in self.pending:
                queue = asyncio.Queue()
                self.pending[key] = queue
            else:
                queue = self.pending[key]

        # If we're the first request, wait for the window and then fetch
        if queue.empty():
            try:
                # Wait for the coalescing window
                await asyncio.sleep(self.window_ms / 1000)

                # Fetch once, then wake everyone
                result = await fetch_fn()

                # Unblock all waiting requests
                while not queue.empty():
                    queue.put_nowait(result)
            finally:
                async with self.lock:
                    self.pending.pop(key, None)
        else:
            result = await queue.get()

        return result

Probabilistic Early Expiration

Instead of waiting for the TTL to expire, you probabilistically refresh the cache before it actually expires. Keys that are frequently accessed get refreshed more often, spreading the load over time rather than having all requests hit at once.

The formula: refresh at ttl - (-ln(random()) * average_ttl * probability). For more on eviction policies that interact with this pattern, see cache eviction policies.

import random
import time

def should_refresh_early(ttl: int, access_frequency: float, random_factor: float = 0.3) -> bool:
    """
    Probabilistic early expiration.
    Higher access frequency = higher chance of early refresh.
    """
    probability = min(1.0, access_frequency * random_factor)
    return random.random() < probability

def get_with_probabilistic_refresh(key: str, fetch_fn, ttl: int = 60):
    value = redis.get(key)
    if value:
        access_count = redis.hincrby("access_counts", key, 1)
        access_frequency = access_count / ttl

        if should_refresh_early(ttl, access_frequency):
            # Asynchronously refresh in background
            asyncio.create_task(refresh_async(key, fetch_fn, ttl))

        return json.loads(value)

    # Cache miss - must wait for fetch
    result = fetch_fn()
    redis.setex(key, ttl, json.dumps(result))
    return result

This works well for read-heavy workloads where some keys are much more popular than others.

Lock-Based Cache Refresh

Instead of letting all requests miss when a key expires, you give one request the “lock” to rebuild the cache while others either return stale data or wait.

import redis
import json
import time
import threading

STALE_TTL = 30  # Keep serving stale data for 30 extra seconds

def get_with_lock_refresh(key: str, fetch_fn, ttl: int = 60):
    cached = redis.get(key)
    if cached:
        value, expires_at = json.loads(cached)
        is_stale = time.time() > expires_at

        if not is_stale:
            return value

        # Stale - try to acquire refresh lock
        lock_key = f"lock:{key}"
        lock_acquired = redis.set(lock_key, "1", nx=True, ex=10)

        if lock_acquired:
            # We got the lock - refresh in background
            def background_refresh():
                try:
                    result = fetch_fn()
                    new_expires = time.time() + ttl
                    redis.setex(key, ttl + STALE_TTL, json.dumps([result, new_expires]))
                finally:
                    redis.delete(lock_key)

            threading.Thread(target=background_refresh, daemon=True).start()

        # Return stale data while refresh happens in background
        return value

    # No cache at all - must fetch synchronously
    result = fetch_fn()
    expires_at = time.time() + ttl
    redis.setex(key, ttl + STALE_TTL, json.dumps([result, expires_at]))
    return result

This is essentially the stale-while-revalidate pattern that Cloudflare, Fastly, and CDNs use — serve stale content while refreshing in the background.

Choosing a Strategy

Combining Approaches

Most production systems layer multiple strategies:

1. Use single-flight at the application layer to deduplicate concurrent requests
2. Add probabilistic early expiration on popular keys to spread load over time
3. Keep a short stale TTL as a safety net for the long tail

async def get_cached_with_layers(key: str, fetch_fn, ttl: int = 60):
    # Layer 1: Single-flight deduplication
    result = await single_flight.get(key, lambda: refresh_with_layers(key, fetch_fn, ttl))
    return result

async def refresh_with_layers(key: str, fetch_fn, ttl: int):
    # Probabilistic early refresh
    if should_probabilistic_refresh(key, ttl):
        asyncio.create_task(background_refresh(key, fetch_fn, ttl))

    # Lock-based refresh if needed
    if is_stale(key):
        await acquire_refresh_lock(key, fetch_fn, ttl)

    return get_cached_or_fetch(key, fetch_fn)

Common Mistakes

Setting TTLs too uniformly: If every key expires at the same time (e.g., all set at application startup), you create synchronized stampedes. Add jitter:

import random
ttl_with_jitter = base_ttl + random.randint(0, int(base_ttl * 0.1))

Ignoring the lock timeout: If your lock acquisition doesn’t have a timeout, a crashed refresh process leaves the lock hanging forever. Always set EX on the lock key.

Serving stale data without bounds: The stale-while-revalidate pattern requires a maximum staleness threshold. Without one, you could serve data that’s hours old.

Monitoring for Stampedes

Watch these signals:

-- Redis: Detect stampede patterns
-- Many identical keys expiring at once
MONITOR | grep -c "EXPIRED"

-- Database: Detect thundering herds
SELECT COUNT(*), DATE_TRUNC('second', created_at) as t
FROM queries
WHERE created_at > NOW() - INTERVAL '10 seconds'
GROUP BY t
HAVING COUNT(*) > 100;

Set alerts on cache hit rate drops — a sudden drop often signals a stampede.

Trade-off Analysis

Real-world Failure Scenarios

Scenario 1: Redis Lock Deadlock After Deploy

What happened: A new deployment changed the lock key prefix from lock: to Lock: (capitalization). The existing lock cleanup logic only matched lowercase lock:. Within 2 hours, 47 stale locks accumulated across production nodes.

Root cause: Lock keys created before the deploy used the old prefix. After deploy, new requests tried to acquire locks with the new prefix but could never clean up the old ones because cleanup logic only searched lock:*.

Impact: Cache stampede on the product_catalog key caused 3400 database queries in 10 minutes. Database CPU hit 95%. Response latency p99 jumped from 12ms to 8,400ms.

Lesson learned: Always include lock timeouts (TTL) as a safety net. Never rely solely on explicit lock release. Add monitoring for stuck locks (WATCHDOG pattern in Redis). Test lock cleanup logic as part of deployment validation.

Scenario 2: Probabilistic Refresh Collisions on Flash Sale

What happened: During a flash sale, a product page cache key had 30-second TTL. The probabilistic early expiration used a beta of 0.3 but the traffic spike meant 500 concurrent requests all hit the cache at the same time before any had a chance to refresh.

Root cause: The probabilistic formula assumed gradual access patterns. During a flash sale, all users arrived simultaneously and the probability calculation never triggered early refresh because TTL was too short and access was too bursty.

Impact: 500 requests all saw cache miss at once, all hit the database, database crashed under load. The stampede lasted 3 minutes until automatic scaling finally caught up.

Lesson learned: Probabilistic early expiration works for gradual traffic patterns, not burst patterns. For flash sales or expected traffic spikes, use proactive cache warming before the event. Combine probabilistic refresh with request coalescing for burst protection.

Scenario 3: Stale Data Served Indefinitely After Bug Fix

What happened: A bug fix changed the product price calculation logic. The cache was set with a 24-hour TTL and stale-while-revalidate of 3600 seconds. After the fix deployed, users saw stale prices for up to 25 hours.

Root cause: The stale-while-revalidate setting allowed the cache to serve stale data for up to 1 hour past expiration while fetching in background. But the background refresh was also failing (the new calculation threw errors on cached data format), so the stale data kept being served from cache without ever being refreshed.

Impact: Customers purchased products at incorrect prices for 25 hours until the cache naturally expired. Estimated $47,000 in overcharges that required manual refund processing.

Lesson learned: Stale-while-revalidate requires monitoring of the refresh success rate. If background refresh fails repeatedly, you must invalidate the key rather than keep serving stale data. Implement a maximum staleness threshold beyond which the cache must return an error or bypass itself.

Real-world Case Study: Twitter’s Cache Stampede Protection

Twitter faces a unique cache stampede challenge: when a celebrity tweets, millions of users simultaneously try to fetch the same data. Their approach combines multiple strategies:

Fan-out on write: When a tweet is published, Twitter proactively pushes the tweet into the timelines of active followers’ caches rather than waiting for cache misses. This is write-time cache population — the fanout-on-write pattern.

Probabilistic early expiration with tenant-aware TTLs: Twitter uses dynamic TTLs based on tweet popularity. High-engagement tweets get shorter TTLs to prevent synchronized expiration. Lower-engagement tweets get longer TTLs.

Deduped fetch windows: During a spike event (like a live tweet from a verified account), Twitter temporarily extends the request coalescing window from 5ms to 50ms to aggregate more requests before making a single database call.

The lesson: for extreme traffic patterns, pre-warming and shorter TTLs beat reactive stampede prevention. If you know an event will cause a thundering herd, warm the cache before the herd arrives.

Interview Questions

ttl_with_jitter = base_ttl + random.randint(0, int(base_ttl * 0.1))

Further Reading

Documentation

• Redis Cache Stampede Prevention — Official documentation on thundering herd mitigation
• Memcached Lock Pattern — Distributed locking implementation for Memcached
• AWS Caching Best Practices — Cloud-native cache strategies including stampede prevention

Papers and Technical References

• Probabilistic Early Expiration Paper — The XFetch algorithm for cache stampede prevention
• ARC: Adaptive Replacement Cache — IBM’s self-tuning cache algorithm combining LRU and LFU

Related Blog Posts

• Caching Strategies — Core caching patterns including cache-aside, write-through, and refresh-ahead
• Cache Eviction Policies — How LRU and LFU interact with stampede prevention
• Distributed Caching — Multi-node cache stampede protection with consistent hashing
• Redis vs Memcached — Stampede prevention implementation differences between engines

Tools

Conclusion

Cache stampede prevention is one piece of a larger caching strategy. For more on caching patterns, read about caching strategies and distributed caching.

For handling the database side of concurrent writes, the locking and concurrency guide covers pessimistic and optimistic approaches.

For understanding how cache stampedes relate to overall database scaling, it is worth knowing how caches fit into the scale-out picture.