We talk about cache hit rates as if the misses are free — as if a 95% hit rate means we’ve solved 95% of the problem. But the miss is where all the interesting cost lives, and it rarely shows up on the dashboard you’re looking at.
A miss isn’t just a slower request. It’s a query that lands on the database at the exact moment ten thousand other misses do, because whatever invalidated one entry probably invalidated the rest. It’s the thundering herd. It’s the tail latency that turns a p50 of 8ms into a p99 of 900ms. It’s the retry storm that follows when the first wave times out.
The lesson I keep relearning: design for the miss, not the hit. Assume the cache is cold. Ask what happens when it all evicts at once. The systems that stay up under load aren’t the ones with the highest hit rates — they’re the ones that degrade gracefully when the hits stop coming.