Cache stampede: 12 misses, one cache fill

Start with the burst you need to absorb

An empty cache can turn a popular read into a burst of identical database queries. Each request sees a miss before the first query finishes. Request coalescing lets those requests share a pending load: one caller starts the work and the others wait for its result. This is often called singleflight or cache stampede protection.

The useful question is where that sharing happens. A lock inside an application process cannot coordinate a different process. A shared value cache does not automatically provide shared ownership of the work needed to populate it.

For this article, Dreamtsoft built and ran a deterministic model with 12 requests for one missing key, distributed evenly across three process labels. Every request arrives before any source operation completes. Uncoordinated loading starts 12 fills. A separate flight registry per process starts three. A common registry starts one. Those are operation counts under an explicit schedule. The model measures no latency and runs no actual distributed cache.

The pattern has an established basis. AWS describes request coalescing as a response to many clients requesting the same uncached resource, including during cold starts. Its discussion also explains why bypassing an unavailable cache can overload the dependency. AWS Builders' Library: caching challenges and strategies.

Read the experiment before choosing a lock

The fixture separates the cache of completed values from the registry of pending fills. Its completion barrier keeps each fill open until the whole request wave has arrived. This removes timing luck from the comparison. Process labels affect registry keys. They do not create operating-system processes.

All requests in these successful cases receive the value belonging to their key. Repeating the comparison with two tenant-specific keys sends 24 requests and produces 24, six and two fills respectively. Distinct tenant values remain distinct. A single global promise would reduce the counter further by sharing work that must stay separate. That would be a correctness bug.

Download the Python model, result table, assertion results and event trace. Put the model in a writable folder and run it with Python 3.10 or later. It uses the standard library and writes its outputs beside the script.

python3 experiment.py

Inspect a trace from a common-registry case: a start event is followed by join events, then one publication resolves the waiting requests. The per-process trace has a start for each process label. No network connection, expiration timer or lease service participates in this experiment.

Give pending work a precise identity and lifecycle

A coalescing key must describe work whose result every joining caller is allowed to receive. For a tenant catalog, that may include tenant identity, catalog version and locale. Derive tenant identity from trusted authorization context. Do not join two reads merely because their URL paths match.

The value-cache key and the flight key should describe the same reusable result. Authorization still runs for each caller. If output depends on a user's permissions, either include the relevant identity in the reusable-result boundary or avoid sharing that output. The fixture checks two explicit tenant keys. It does not test an application's access-control implementation.

In a concurrent implementation, creation of the pending entry must be atomic within the chosen coordination scope. The winning caller should recheck the value cache before loading: another fill may have completed between its first miss and ownership acquisition. Publish a usable result before removing the pending entry. On failure, complete the waiting calls with the defined error and remove only the entry this owner controls.

Go's singleflight package expresses a related contract: calls with the same key in a Group share an execution that is still in progress. The duplicate receives its result. This is duplicate-call suppression. A completed value still needs a separate cache if later calls should reuse it. Go singleflight documentation.

Decide whether process-local protection is enough

Three fills can be an acceptable result. If the source can absorb one fill per application process during a cold start, local coalescing may remove the immediate pressure without adding a distributed ownership service. Test the burst at the largest expected fleet size, including overlap between old and new deployments.

The model's common registry is an idealized boundary. Implementing equivalent coordination across a fleet introduces failure cases it does not contain: a worker can die while holding ownership, a lease can expire while its query still runs and a disconnected owner can return late. A lease timeout alone cannot guarantee that only one query is executing. Define how stale owners are prevented from overwriting an accepted result, where the storage system supports that protection.

Prefer a bounded source-load objective over an unconditional promise of exactly one fill. Specify how many concurrent fills the dependency can tolerate, what callers do when ownership cannot be obtained and how long they may wait. If coordination fails, allowing every waiter to bypass it recreates the original burst.

Coalescing does not solve value freshness. A shared load can still publish an old snapshot after a write. Review the separate cache-aside stale-write race when the cached entity can change during a fill.

Treat failure and cancellation as part of the contract

In the failure case, the model starts one fill for 12 callers and returns 12 errors. It leaves no pending entry and stores no successful value. The next wave can create a new fill, which succeeds for all 12 callers. A subsequent warm-cache wave starts zero fills. These assertions catch a registry that retains a completed failure forever.

Immediate cleanup does not provide a retry policy. If every failed caller retries at once, each subsequent wave can still hit the source. Set a retry budget for the load path and decide which outcomes may be cached briefly. A temporary outage and an authoritative missing record have different meanings. Do not turn one into the other merely to reduce requests. The request deadline and retry budget model provides a separate way to reason about that limit.

A waiting request also has its own deadline. One caller abandoning its wait should not automatically cancel work still needed by others. Conversely, a source operation must not run forever after its audience disappears. Choose an operation deadline and an explicit policy for the last waiter leaving. Cancellation behavior is an integration check proposed here, not an executed result of the fixture.

Add expiry controls without confusing their jobs

TTL jitter spreads expiration times across different entries. It does not merge requests that have already missed the same entry. Refreshing before expiry can reduce cold reads, but concurrent refresh workers still need coordination if duplicate work is expensive.

Serving a stale value while refresh runs changes the freshness contract. For data that permits it, define the maximum acceptable age and the response after that limit. A caller requesting a current permission decision should not inherit the same stale-data policy as a catalog preview. AWS describes soft and hard TTLs as one way to retain a usable older value during downstream trouble. AWS cache expiration guidance.

These choices address different failure paths. Start with the source-load limit, then decide whether callers can wait, receive an older value or be rejected. A cache hit ratio by itself cannot tell you whether one expensive key is starting duplicate fills.

Take a cold-cache acceptance test to your service

Use the fixture as a review aid, then test the real runtime. The following checks are proposed acceptance work. This article does not report that they passed against a deployed service.

• Hold one source load open, issue overlapping reads and count actual source operations across every application instance. Confirm that the count matches the intended coordination scope.
• Request two tenant-specific versions of the same entity. Check both returned values and the keys used for pending work.
• Fail the loader, release all waiters and send another wave. Verify cleanup, retry limits and whether any error cache has the intended meaning.
• Cancel a waiting request, then terminate the owner separately. Record what happens to remaining waiters and late source responses.
• Repeat with the cache unavailable. Confirm that fallback traffic stays within the dependency's load budget.

Record fills started, waiters joined, fill failures and wait duration alongside cache hits. Aggregate by a bounded operation category. Raw cache keys or tenant IDs can make metric labels grow with customer data. Set the rollout gate against actual source traffic during a cold start, because that is the pressure this change is intended to control.

Sources

Documentation checked September 9, 2026.

1. AWS Builders Library: caching challenges and strategies
2. Go: singleflight package