Skip to content

Invalidating Caches Across Async Workers

An in-process cache is the fastest cache there is — 0.33 µs per hit — and the moment you run more than one worker it becomes N independent copies of the truth. One worker handles the update and drops its entry; the other seven keep serving what they cached, for as long as their TTL allows. Redis pub/sub fixes the latency (measured: 0.2 ms to reach four workers) but not the reliability, because a worker that is reconnecting when the message goes out never learns, and nothing replays it. This guide covers the four strategies, what each guarantees, and the ordering rule that invalidates half of them if you get it wrong.

Prerequisites

Four ways to stop workers serving stale data A grid of 4 rows by 2 columns. Four ways to stop workers serving stale data strategy staleness window fails when a short TTL up to the TTL never: it is self-healing pub/sub invalidation sub-millisecond, measured 0.2 ms a subscriber is disconnected versioned keys none: old keys are unreachable never, but keys accumulate write-through to Redis none for shared reads local caches still need one of the above Only the TTL and versioning are self-correcting; pub/sub alone leaves a worker permanently stale.

1. Start with a TTL short enough to be your worst case

Before any messaging, decide how stale the data may be, and make that the TTL. A 30-second TTL means "no worker serves data more than 30 seconds old", unconditionally, with no moving parts. Everything else in this guide is an optimisation on top of that guarantee, not a replacement for it.

The reason is simple: every push-based mechanism can miss. A TTL cannot — it is the only self-healing invalidation there is. Services that replace the TTL with pub/sub discover this during an incident, when one instance serves a deleted user's profile until it happens to be restarted.

Verify: every local cache entry has an expiry, and the longest one is a number the product can live with.

2. Publish invalidations, and measure the latency

For anything where 30 seconds is too long, broadcast the key:

async def invalidate(redis_client, key: str) -> None:
    await redis_client.publish("cache:invalidate", key)


async def subscriber(redis_client, local_cache, stop: asyncio.Event) -> None:
    pubsub = redis_client.pubsub()
    await pubsub.subscribe("cache:invalidate")
    try:
        while not stop.is_set():
            message = await pubsub.get_message(ignore_subscribe_messages=True, timeout=0.05)
            if message:
                local_cache.drop(message["data"].decode())
    finally:
        await pubsub.unsubscribe("cache:invalidate")
        await pubsub.aclose()

Measured with four subscriber workers: all four dropped the key, with a propagation latency of 0.1–0.2 ms. That is fast enough that the staleness window is effectively the network.

Two implementation details. The subscriber needs its own connection — a pubsub() connection cannot serve normal commands, exactly as with a LISTEN/NOTIFY listener. And the subscriber task belongs to the lifespan, so it is cancelled cleanly at shutdown.

Verify: updating a value on one worker drops it on all of them within a millisecond.

An invalidation message in flight 5 stages from write the source to next read refills. An invalidation message in flight write the source database or Redis publish the key after the commit workers receive it in about 0.2 ms local entry dropped not repopulated next read refills from the new value Publishing before the commit is the classic bug: workers refill from the old value.

3. Publish after the commit, never before

The ordering bug is easy to write and hard to see:

# wrong
await invalidate(redis_client, key)                    # workers drop it...
await conn.execute("UPDATE users SET ... WHERE id = $1", user_id)   # ...and refill the OLD row

Between the publish and the commit, every worker that reads that key repopulates its cache from the value you are in the middle of replacing — and now they are stale with no further message coming. Publishing after the commit is correct:

async with conn.transaction():
    await conn.execute("UPDATE users SET ... WHERE id = $1", user_id)
await invalidate(redis_client, key)                    # after the data is visible

For the version where even that gap matters, write the invalidation into the outbox alongside the update, so it is published exactly when the change becomes visible and cannot be lost if the process dies between the two statements.

Verify: a concurrent reader during an update ends up with the new value, not the old one.

Why the publish comes after the commit 3 lanes over time. Why the publish comes after the commit writer transaction open COMMIT publish too early publish refills the old value correct publish worker refills new value time → Between an early publish and the commit, every reader repopulates the value you are replacing.

4. Know what a missed message costs

Pub/sub is at-most-once, and this is the measurement that matters:

with one subscriber down: values ['old', None, None, None]
-> worker 0 keeps serving stale data

Three workers dropped the key; the one whose subscriber had stopped kept its stale copy, and a reconnect does not replay anything. Whatever the cause — a restart, a network blip, a slow consumer — that worker serves stale data until its TTL expires.

Which is exactly why step 1 exists. Pub/sub reduces the typical staleness from the TTL to a millisecond; the TTL bounds the worst case. Anyone reasoning about correctness should use the TTL, and anyone reasoning about user experience should use the pub/sub latency.

Add a resubscribe-and-flush rule for robustness: when the subscriber reconnects, clear the entire local cache rather than assuming it is still valid. That converts "stale until the TTL" into "stale until the reconnect", which is usually much shorter.

Verify: kill and restart a worker's subscriber and confirm its local cache is flushed on reconnect.

What a missed message costs 2 columns contrasting pub/sub only, versioned keys. What a missed message costs pub/sub only permanently stale measured: 3 workers dropped the key the disconnected one kept "old" nothing corrects it before the TTL reconnect does not replay versioned keys self-correcting the version is part of the key a bumped version misses everywhere no message to miss old entries expire unused Combine them: pub/sub for latency, a bounded TTL so a missed message cannot last forever.

5. Prefer versioned keys where the shape allows

The strategy with no messages and no missed invalidations is to make the old key unreachable:

version = await redis_client.get(f"ver:user:{user_id}") or 1
key = f"app:user:{user_id}:v{version}"

# on update
await redis_client.incr(f"ver:user:{user_id}")         # every worker now misses

Verified: incrementing the version moved the key from user:3:v1 to user:3:v2, so every worker's cached entry became unreachable without anyone being told. Old entries are never read again and expire on their own.

The cost is a read of the version — which can itself be cached with a short TTL, trading a small staleness window for the round trip. A coarser variant works well for grouped data: one version per tenant, per table or per deploy, bumped when anything in that group changes. That is the same mechanism as bumping a key prefix on a format change, applied at runtime.

Where it does not fit is data with many independent fine-grained keys, where a version lookup per key is as expensive as the cache read it protects.

Verify: after an update, no worker can construct the old key.

Verification

Cross-worker invalidation is correct when:

  • Every entry has a TTL that bounds the worst-case staleness.
  • Invalidations are published after the commit, never before.
  • The subscriber has its own connection and is owned by the lifespan.
  • A reconnect flushes the local cache rather than trusting it.
  • Versioned keys are used where the data shape allows.
  • Staleness is measurable: a metric or a log line when a worker drops an entry.

Pitfalls & edge cases

  • Relying on pub/sub alone. A disconnected worker is stale until it restarts — verified.
  • Publishing inside the transaction. Readers refill from the pre-commit value.
  • Invalidating on the wrong key shape. A message for user:42 does not touch user:42:profile; invalidate prefixes deliberately.
  • Flushing the whole cache on every message. Correct but expensive; a storm of invalidations becomes a storm of misses.
  • Subscribing on the shared client. The connection can no longer run ordinary commands.
  • Version counters without expiry. They accumulate one key per entity; keep them in a table or give them a long TTL.

Frequently Asked Questions

How do I invalidate in-process caches across multiple workers?

Publish the invalidated key on a Redis channel that every worker subscribes to, and have each subscriber drop the entry from its local cache. Measured with four workers, propagation took 0.1–0.2 ms. Keep a TTL as well, because pub/sub messages can be missed.

Is Redis pub/sub reliable enough for cache invalidation?

Only as an optimisation. It is at-most-once: a worker whose subscriber is disconnected never receives the message and never gets a replay — verified, that worker kept serving the stale value while three others dropped it. The TTL is what bounds the damage.

Should I invalidate the cache before or after the database write?

After the commit. Invalidating first opens a window in which readers repopulate the cache from the pre-update value and are then stale with no further message coming. For the strongest version, write the invalidation into a transactional outbox with the update.

What are versioned cache keys?

Keys that embed a version number — app:user:42:v7 — which is incremented when the entity changes. Every worker then misses without being told anything, so there is no message to lose. Old entries become unreachable and expire on their own.

How stale can my cache be after an update?

In the typical case, the pub/sub latency: about 0.2 ms in these measurements. In the worst case, the TTL — because any worker that missed the message keeps its copy until then. Quote the TTL when reasoning about correctness and the pub/sub latency when reasoning about user experience.