Cache Strategies in Distributed Systems
Caching is one of the most powerful techniques in distributed systems. It dramatically reduces latency, protects databases, and allows systems to scale to millions of users.
One of the most common problems is the Thundering Herd (Cache Stampede), where many requests simultaneously try to recompute expired cache entries, overwhelming backend services.
Why Basic TTL Caching Is Not Enough
The simplest cache strategy is TTL-based caching.
If the cache entry exists → cache hit
If not → fetch from database
This works well most of the time. However, it fails under synchronized cache expiration.
This phenomenon is called cache stampede, a variant of the thundering herd problem.
These spikes can crash databases in seconds.
Strategies to mitigate this issue
1. TTL Jitter (Randomized Expiration)
If all cache entries share the same TTL, then all entries expire together. All requests recompute simultaneously. The Solution: Add Randomness
Instead of fixed TTL: TTL = 60s + random(0–20s).
This spreads recomputation over time.
2. Probability-Based Early Expiration
This technique is sometimes called, Probabilistic Early Re-computation. It prevents sudden expiration bursts. Instead of waiting until TTL = 0, the system sometimes refreshes the cache early. If TTL is close to expiring, some requests probabilistically recompute it.
3. Mutex / Cache Locking
Another common solution is cache locking. 10,000 requests Cache miss can cause 10,000 DB queries. With Mutex, Only one request recomputes the cache.
Systems typically use: Redis locks, Memcached locks, ZooKeeper locks.
4. Stale-While-Revalidate (SWR)
This technique allows serving stale data temporarily. Users shouldn't wait while cache refreshes. Serve stale data immediately, refresh in background. Users see low latency, while cache refresh happens asynchronously.
5. Cache Warming / Pre-Warming
Cache warming means preloading cache before traffic arrives. Before a large sale, companies like Flipkart preload product caches.
Tradeoffs:
Every caching strategy involves tradeoffs.
| Factor | Meaning |
|---|---|
| Freshness | How up-to-date the data is |
| Latency | Response time |
| Consistency | Whether all users see the same data |
Example:
SWR prioritizes: Low latency But allows slightly stale data.
Strict Consistency: Financial systems may prefer: Strong consistency Even if latency increases.
Choosing the Right Strategy
| Scenario | Recommended Strategy |
|---|---|
| High read traffic | TTL jitter |
| Heavy cache rebuild cost | Mutex locking |
| CDN-like workloads | Stale-While-Revalidate |
| Predictable spikes | Cache warming |
| Extreme traffic | Probabilistic expiration |
Key Takeaways
Basic TTL caching works only at small scale.
Large systems require:
randomized expiration
background refresh
cache locks
probabilistic refresh
cache warming
These techniques prevent:
database overload
latency spikes
cascading failures
