Skip to content

Async Caching & Deduplication in Python

Caching in an async service has one property that makes it different from caching anywhere else: thousands of requests can be in flight simultaneously, so a single expired key produces a single moment in which every one of them misses. Measured with 50 concurrent requests for one key whose rebuild takes 200 ms, the unprotected cache made 50 origin calls; adding one lock per key made it 1. The concurrency that makes asyncio fast is the same concurrency that turns an expiry into a stampede, and deduplication — not storage — is the part that needs designing.

The rest follows from where the value lives. A process-local cache answers in 0.4 µs and exists once per worker, so eight workers hold eight versions of the truth. Redis answers in about a millisecond when reads are batched, is shared, and survives a deploy — but adds a network round trip to a path whose whole purpose is to be fast. Most real services end up with both, plus an invalidation mechanism whose typical latency is a fraction of a millisecond and whose worst case is the TTL. This section covers each layer with its measurements; the parent section, Concurrent Execution & Worker Patterns, covers the concurrency primitives underneath.

Scope of this section:

  • Coalescing concurrent misses so one rebuild serves every waiter.
  • Stampede protection: locks, early expiry and stale-while-revalidate, measured.
  • Building a TTL cache decorator whose cancellation semantics are correct.
  • Redis as a shared cache: pooling, batching, TTL rules and serialisation.
  • Invalidating per-process caches across workers, and what a missed message costs.

Architectural principles

  • Deduplicate before you optimise storage. One lock per key took 50 concurrent misses to a single origin call. No amount of faster storage fixes a stampede, and nothing else you do matters as much.
  • State a staleness budget. Every cache serves data that is out of date by up to some amount. Write that number down; it becomes the TTL, and every push-based invalidation is an optimisation on top of it.
  • Bound every cache. A dictionary with no maxsize is a memory leak keyed by user input. Eviction policy is a design decision, not an afterthought.
  • The cache must never fail a request. A Redis timeout is a miss, not a 500. Set socket timeouts, catch RedisError, count it, and fall through to the origin.
  • Nobody owns an in-flight fill. If the first caller's cancellation kills the shared call, a timeout on one request fails every other request waiting for that key — a bug that only appears under the conditions you most need the cache.
Three tiers, three orders of magnitude 4 stacked layers from L1: in process to invalidation. Three tiers, three orders of magnitude L1: in process 0.4 µs per hit, measured per worker: N copies of the truth L2: Redis 1.3 ms for 1,000 keys batched shared, survives a deploy origin tens of milliseconds the thing you are protecting invalidation pub/sub in 0.2 ms plus a TTL, because messages are lost Verified end to end: 22,000 reads over 50 keys reached the origin 50 times.

Execution model: one loop, one key, many waiters

The unit that matters is the key, not the request. On one event loop, every request for a given key at a given moment is in one of three states: it found a live entry, it is waiting on a fill that another task started, or it is the fill. A correct async cache makes the third state happen exactly once, and the measurement that proves it is the ratio of origin calls to requests — in the integrated example below, 50 origin calls for 22,000 reads.

That framing also explains the cancellation rule. If the fill runs inside the first caller's task, then that caller's timeout cancels work that N others are waiting for, and they receive a CancelledError for a request they never cancelled. Running the fill in a task the cache owns, with callers awaiting asyncio.shield(task), makes each caller's cancellation local — which is the difference between a cache that degrades gracefully under load and one that amplifies a single slow request into N failures.

The second consequence is that per-process caches multiply. Eight uvicorn workers mean eight L1 caches, eight cold starts after a deploy, and eight copies that must be invalidated independently. Redis collapses that to one copy at the cost of a round trip, which is why the two-tier arrangement — a small hot L1 in front of a shared L2 — is so common: L1 absorbs the request rate, L2 absorbs the misses, and only L2's misses reach the origin.

One read through a two-tier cache 5 stages from L1 lookup to publish the key. One read through a two-tier cache L1 lookup sub-microsecond L2 lookup one round trip one origin call others coalesce write both tiers TTL with jitter publish the key other workers drop L1 Every layer below the first is protected by deduplication at the layer above it.

Pattern catalogue

Coalesce concurrent misses

task = inflight.get(key)
if task is None:
    task = asyncio.create_task(compute(key))           # owned by the cache, not a caller
    inflight[key] = task
    task.add_done_callback(lambda _t, k=key: inflight.pop(k, None))
return await asyncio.shield(task)

Measured: 100 concurrent calls, 1 real call, and cancelling one caller left the others unaffected. See building an async TTL cache decorator.

Serve stale while refreshing

if float(fresh_until or 0) < time.time() and not lock_for(key).locked():
    task = asyncio.create_task(refresh(key))           # nobody waits for this
    refreshers.add(task)
    task.add_done_callback(refreshers.discard)
return value                                           # stale, but immediate

Measured p50 of 0.76 ms with refreshes happening throughout; only the cold start blocked. See preventing cache stampedes.

Batch every Redis access

values = await redis_client.mget(keys)                 # 1.3 ms for 1,000 keys
async with redis_client.pipeline(transaction=False) as pipe:
    for key, value in items.items():
        pipe.set(key, value, ex=jittered(ttl))
    await pipe.execute()                               # 12 ms for 1,000 writes

Individually those were 61 ms and 69 ms. See caching with redis.asyncio.

Invalidate across workers, after the commit

async with conn.transaction():
    await conn.execute("UPDATE users SET ... WHERE id = $1", user_id)
await redis_client.publish("cache:invalidate", key)    # never before the commit

Propagation to four workers took 0.1–0.2 ms, and a worker whose subscriber was down kept its stale copy — which is why the TTL still matters. See invalidating caches across async workers.

Always write a TTL

await redis_client.set(key, value, ex=int(ttl * random.uniform(0.9, 1.1)))

A plain SET clears an existing expiry, leaving the key cached forever — verified, TTL returned -1. The jitter stops keys written together from expiring together.

Cache absences too

if result is None:
    await redis_client.set(key, MISS_SENTINEL, ex=NEGATIVE_TTL)   # seconds, not minutes
    raise NotFound(key)

A key that does not exist is an uncached path straight to the origin, repeated for every request — which is what a scan of guessed identifiers produces. A short negative TTL stops that without keeping a wrong answer around long enough to matter once the key appears.

The decisions every cache makes, explicitly or not A grid of 5 rows by 2 columns. The decisions every cache makes, explicitly or not decision safe default measured effect concurrent misses coalesce into one call 100 callers -> 1 call staleness a TTL you can defend bounds every other mechanism invalidation pub/sub plus the TTL 0.2 ms typical, TTL worst case size an explicit maxsize unbounded caches are leaks cache errors treat as a miss never fail a request on the cache A cache without a stated staleness budget is a correctness decision nobody made.

Resource boundaries

Resource What consumes it How to size and bound it
L1 memory Entries times their size, per worker An explicit maxsize with LRU eviction
Lock or in-flight tables One entry per key ever requested Remove on completion; bound like the cache
Redis connections Concurrent commands max_connections; choose failing vs queueing deliberately
Redis memory Keys times size, minus expiry TTLs on every write; a maxmemory-policy that evicts
Origin capacity Cache misses, not requests Deduplication; a stampede is an origin outage
Staleness The TTL, not the pub/sub latency State the budget; invalidation only improves the typical case
Background refreshes One task per soft-expired key Track them in a set; cancel at shutdown

The origin row is the one that turns a cache into an incident: a cache is a load-shedding device for the origin, and the only number that matters is misses per second, not requests per second. A service whose request rate doubles but whose miss rate is unchanged has not increased its load on anything downstream; a service whose request rate is flat while a hot key expires can take its database down in one second. The staleness row is the same idea applied to correctness rather than capacity — and both are properties you set deliberately at write time, in the TTL, rather than properties that emerge from how the cache happens to behave.

Integrated production example

A two-tier cache with everything above: an LRU L1, a shared Redis L2, single-flight fills owned by the cache, background refresh when the L2 entry is soft-expired, pub/sub invalidation between workers, and jittered TTLs.

import asyncio
import random
import time
from collections import OrderedDict

import redis.asyncio as redis

L1_MAX, L1_TTL, L2_TTL, SOFT = 1000, 5.0, 60, 30
stats = {"l1": 0, "l2": 0, "origin": 0, "coalesced": 0, "invalidated": 0}


class TwoTierCache:
    def __init__(self, client: redis.Redis, origin):
        self.client, self.origin = client, origin
        self.l1: OrderedDict[str, tuple[float, bytes]] = OrderedDict()
        self.inflight: dict[str, asyncio.Task] = {}
        self.refreshers: set[asyncio.Task] = set()

    def _l1_get(self, key):
        hit = self.l1.get(key)
        if hit and hit[0] > time.monotonic():
            self.l1.move_to_end(key)
            return hit[1]
        return None

    def _l1_set(self, key, value):
        self.l1[key] = (time.monotonic() + L1_TTL, value)
        self.l1.move_to_end(key)
        while len(self.l1) > L1_MAX:
            self.l1.popitem(last=False)                       # bounded, always

    async def get(self, key: str) -> bytes:
        value = self._l1_get(key)
        if value is not None:
            stats["l1"] += 1
            return value
        task = self.inflight.get(key)
        if task is not None:
            stats["coalesced"] += 1
            return await asyncio.shield(task)                 # join the existing fill
        task = asyncio.create_task(self._fill(key))           # owned by the cache
        self.inflight[key] = task
        task.add_done_callback(lambda _t, k=key: self.inflight.pop(k, None))
        return await asyncio.shield(task)

    async def _fill(self, key: str) -> bytes:
        pipe = self.client.pipeline(transaction=False)
        pipe.get(key)
        pipe.ttl(key)
        value, ttl = await pipe.execute()                     # one round trip for both
        if value is not None:
            stats["l2"] += 1
            self._l1_set(key, value)
            if ttl < SOFT:                                    # soft-expired: refresh behind the read
                task = asyncio.create_task(self._refresh(key))
                self.refreshers.add(task)
                task.add_done_callback(self.refreshers.discard)
            return value
        return await self._refresh(key)

    async def _refresh(self, key: str) -> bytes:
        stats["origin"] += 1
        value = await self.origin(key)
        await self.client.set(key, value, ex=int(L2_TTL * random.uniform(0.9, 1.1)))
        await self.client.publish("cache:invalidate", key)    # other workers drop their L1
        self._l1_set(key, value)
        return value

    async def subscribe(self, stop: asyncio.Event) -> None:
        pubsub = self.client.pubsub()                         # its own connection
        await pubsub.subscribe("cache:invalidate")
        try:
            while not stop.is_set():
                message = await pubsub.get_message(ignore_subscribe_messages=True, timeout=0.05)
                if message:
                    self.l1.pop(message["data"].decode(), None)
                    stats["invalidated"] += 1
        finally:
            await pubsub.unsubscribe("cache:invalidate")
            await pubsub.aclose()

Exercised over 50 keys with an origin that takes 50 ms: 2,000 cold concurrent reads completed in 0.07 s, then 20,000 sequential warm reads in 0.01 s at p50 0.4 µs and p99 0.8 µs. The counters read {'l1': 19958, 'l2': 42, 'origin': 50, 'coalesced': 1950, 'invalidated': 50} — a 99.5% L1 hit rate, and 50 origin calls for 22,000 reads, one per distinct key.

Diagnostic Hook — is the cache helping or hiding?

Four numbers. Hit rate per tier, not overall: an L1 hit rate of 99% with an L2 hit rate of 10% means your L2 is doing nothing but adding a round trip. Origin calls per second — the number the cache exists to reduce, and the one that spikes during a stampede while the request rate looks normal. Fill latency p99, which is what a cache miss costs a user, and the reason stale-while-revalidate exists. Cache error rate, counted separately from misses: a Redis that is timing out looks like a cold cache in every other metric, and the two need very different responses. Alert on origin calls per second rather than on hit rate, and on cache errors at any non-trivial level.

Which cache does this value belong in? A decision on How is this value used with 3 outcomes. Which cache does this value belong in? How is this value used? small, hot, tolerant of staleness in-process 0.4 µs, per worker expensive, shared, must agree Redis one copy, one fill changes faster than it is read do not cache it invalidation will dominate The third branch is the one teams skip; a cache with a 5% hit rate is pure overhead.

Failure modes

Failure mode Root cause Detection Fix
Origin overwhelmed when a key expires No deduplication of concurrent misses Origin calls spike, requests do not One lock or in-flight task per key
One timeout fails many requests The first caller owns the fill CancelledError in callers that did not cancel Fill in a cache-owned task, callers shield
Memory grows until the pod dies Unbounded cache or lock dictionary RSS tracks distinct keys seen maxsize with LRU eviction
Stale data after an update Per-worker caches, no invalidation One worker serves old values Publish after commit, plus a TTL
Permanently stale on one worker Missed pub/sub message One instance differs from the rest The TTL; flush L1 on resubscribe
Keys never expire A plain SET cleared the TTL TTL returns -1 Always pass ex=, or keepttl=True
Requests fail when Redis is slow Cache errors treated as fatal 5xx correlated with Redis latency Timeouts, catch RedisError, treat as a miss
Everything expires at once Identical TTLs written together A periodic spike in origin calls Jitter every TTL

Frequently Asked Questions

How do I stop concurrent requests all rebuilding the same cache entry?

Keep an in-flight map keyed the same way as the cache: the first miss starts a task, later callers await that task. Measured, 50 concurrent misses went from 50 origin calls to 1. Make the task owned by the cache rather than by the first caller, so one caller's cancellation does not kill the shared fill.

Should an async service cache in process or in Redis?

Usually both. An in-process cache answers in about 0.4 µs but exists once per worker; Redis is shared and survives deploys but costs a round trip. A small L1 in front of a shared L2 absorbs the request rate locally while keeping origin fills to one per key across the fleet.

How stale can cached data be?

Up to the TTL, always — that is the only self-healing bound. Pub/sub invalidation reduced the typical staleness to 0.2 ms in testing, but a worker whose subscriber was disconnected kept serving old data with no replay. Quote the TTL for correctness and the invalidation latency for user experience.

What should happen when Redis is unavailable?

The request should still succeed. Set socket timeouts, catch RedisError, count it as a distinct metric and fall through to the origin. A cache that can fail a request is a new single point of failure in front of the one you were trying to protect.

How do I know whether a cache is worth keeping?

Look at origin calls per second before and after, not at hit rate alone. A cache with a low hit rate adds latency, memory and invalidation complexity while removing almost no load — and data that changes faster than it is read should not be cached at all.