Cache stampede and the thundering herd

What happens when a hot key expires and thousands of requests miss at once.

A cache protects the database only while entries are present. Consider the front page of your delivery app: one key, home:feed, read 10,000 times per second, rebuilt by a query that takes 200 ms. When that key expires, every request arriving in the next 200 ms misses, and each one starts the same expensive query. That is about 10,000 × 0.2 = 2,000 identical queries hitting the database at once. A database sized for the cached load can fall over, and slower responses mean even more concurrent misses. This is a cache stampede, or thundering herd.

The root problem is that nothing coordinates the missers: each thinks it is the only one. The fixes all add coordination or remove the synchronised expiry. They can be combined, and which you pick depends on how expensive the rebuild is and how much staleness you can show.

def get_feed():
    value, fresh = cache.get_with_soft_ttl("home:feed")
    if value is not None and fresh:
        return value
    if cache.set_if_absent("lock:home:feed", ttl=5):
        value = rebuild_feed()           # only one request does this
        cache.set("home:feed", value, ttl=60, soft_ttl=45)
        return value
    return value if value is not None else wait_and_retry()