What to cache: hit ratio math

Decide what deserves cache space and work out what a cache really saves.

Caching everything is neither possible nor wise. Cache an item when it is read often, changes rarely, costs a lot to compute or fetch, and tolerates some staleness. Good candidates are restaurant menus, public profile pages and computed rankings. Poor candidates are data read once, data that changes constantly, and anything that must be perfectly current such as an account balance.

The value of a cache is measured by its hit ratio, and the effect is bigger than people expect. Say a cache read takes 1 ms and a database read takes 50 ms. Average latency is h × 1 + (1 − h) × 50 ms. At 80 percent hits that is 0.8 + 10 = 10.8 ms. At 90 percent it is 0.9 + 5 = 5.9 ms. At 99 percent it is 0.99 + 0.5 = 1.49 ms. Going from 90 to 99 percent looks like a small step but it cuts latency by about four times.

The database effect matters even more. If 10,000 requests per second arrive and the hit ratio is 90 percent, the database sees 1,000 per second. At 99 percent it sees only 100 per second, one tenth. This is why a small improvement in hit ratio can let you run a much smaller database, or survive a traffic spike that would otherwise topple it.

Worked example: sizing the benefit

Step 1 of 3

Start

Traffic is 10,000 reads per second. A database node handles 6,000 per second, so with no cache you need at least 2 nodes and are close to the edge.

A scan hurts FIFO more than LRU

Replay a hot key A mixed with one-time keys using FIFO, then switch the policy to LRU. Compare hits.

Capacity: 3

Policy

TTL: none

Apending
Bpending
Apending
Cpending
Apending
Dpending
Apending
Epending
Apending
Fpending
Apending
Bpending

Cache (next to evict ← → most recent)

Press Step to begin.

So far: 0 hits / 0 misses

Question 1

Cache read is 1 ms, database read is 50 ms. What is the average latency at a 90 percent hit ratio?

Question 2

10,000 reads/s, hit ratio rises from 90 to 99 percent. What happens to the database read rate?

Your turn

Choose three things worth caching and three that are not in a ticket-booking site, and say why in a few words each.

Stuck? Ask how often it is read, how often it changes and how bad stale data is.

What to remember

Cache what is read often, changes rarely, is expensive to produce and can tolerate staleness.
Average latency is h × cache + (1 − h) × database; small hit-ratio gains cause big drops in database load.
If the system cannot survive without the cache, the cache is part of capacity and its failure needs a plan.
Measure hit ratio per cache and per key type.