Skip to content

Implementing the Single-Flight Pattern for Duplicate Calls

A popular product page goes viral, its cached price expires, and within the same 50 milliseconds three hundred requests find the cache empty and each asks the pricing service for the same number. The pricing service, sized for a trickle of cache misses, falls over; its errors make more requests miss the cache; the incident has begun. The underlying waste exists even without an incident: an async service happily runs the same expensive call dozens of times concurrently because nothing tells the second caller that the first one is already on its way. Single-flight — the name comes from Go's singleflight package — fixes that at the call site: while a call for a given key is in flight, every other caller for that key waits for the same result instead of starting its own. This guide implements it on asyncio tasks, shares failures without caching them, keeps one caller's cancellation from cancelling the call everyone else is waiting on, and combines it with a short-lived cache.

Prerequisites

Downstream calls for a burst of identical requests 3 bars comparing 100 requests, no coordination with the others. Downstream calls for a burst of identical requests 100 requests, no coordination 100 calls 100 requests, single-flight 1 call 300 requests in 3 bursts, + 200 ms cache 1 call Measured with the examples on this page: a 50 ms loader, bursts 50 ms apart. Coalescing removes duplicates that overlap; the cache removes the ones that follow.

1. Measure the duplicate work

Start with the failure. A hundred concurrent requests for one cold key each call the loader, and the loader's call count is the number of requests, not the number of distinct keys.

import asyncio

calls = 0


async def load_profile(user: str) -> dict:
    global calls
    calls += 1
    await asyncio.sleep(0.05)                      # the expensive downstream call
    if user == "bad":
        raise LookupError(user)
    return {"user": user}


async def main() -> None:
    results = await asyncio.gather(*(load_profile("u1") for _ in range(100)))
    print(len(results), "results from", calls, "downstream calls")   # 100 from 100


asyncio.run(main())

In production the same measurement is a metric: downstream call rate per distinct key. A ratio far above one during bursts means callers are duplicating work that could be shared.

Verify: the script reports 100 downstream calls for 100 identical requests.

2. Coalesce callers onto one task per key

Keep a dictionary from key to the in-flight task. The first caller creates the task; later callers find it and await the same task. When the task finishes, a done-callback removes the key so the next burst starts a fresh call rather than reusing a stale result.

import asyncio
from collections.abc import Awaitable, Callable, Hashable
from typing import TypeVar

T = TypeVar("T")


class SingleFlight:
    """Coalesce concurrent calls with the same key into one execution."""

    def __init__(self) -> None:
        self._inflight: dict[Hashable, asyncio.Task] = {}
        self._waiters: dict[Hashable, int] = {}

    async def do(self, key: Hashable, fn: Callable[[], Awaitable[T]]) -> T:
        task = self._inflight.get(key)
        if task is None:                                   # no await between lookup and insert
            task = asyncio.create_task(fn(), name=f"singleflight:{key}")
            self._inflight[key] = task
            self._waiters[key] = 0
            task.add_done_callback(lambda _t, k=key: self._forget(k))
        self._waiters[key] += 1
        try:
            return await asyncio.shield(task)              # see step 4
        except asyncio.CancelledError:
            if not task.done():
                self._waiters[key] -= 1
                if self._waiters[key] == 0:
                    task.cancel()                          # the last interested caller left
            raise

    def _forget(self, key: Hashable) -> None:
        self._inflight.pop(key, None)
        self._waiters.pop(key, None)


async def main() -> None:
    global calls
    calls = 0
    flight = SingleFlight()
    results = await asyncio.gather(*(flight.do("u1", lambda: load_profile("u1")) for _ in range(100)))
    print(len(results), "results from", calls, "downstream call")   # 100 from 1
    print("same object:", results[0] is results[99])


asyncio.run(main())

The lookup and the insert happen with no await between them, so on one event loop two callers can never both decide they are first. Passing a zero-argument callable rather than a coroutine object matters: creating the coroutine eagerly for every caller would leave 99 never-awaited coroutines and a warning for each.

Verify: 100 results from one downstream call, and every caller receives the identical object — so treat shared results as read-only, or copy before mutating.

3. Share failures, but never cache them

When the shared call raises, every waiter should see that exception: they asked the same question and got the same answer. But the failure must not be remembered — the done-callback removes the key whether the task succeeded or failed, so the next caller retries.

import asyncio


async def main() -> None:
    global calls
    flight = SingleFlight()

    calls = 0
    outcomes = await asyncio.gather(
        *(flight.do("bad", lambda: load_profile("bad")) for _ in range(5)),
        return_exceptions=True,
    )
    print(calls, [type(o).__name__ for o in outcomes])     # 1 ['LookupError', ...]

    calls = 0
    try:
        await flight.do("bad", lambda: load_profile("bad"))
    except LookupError:
        print("a later caller retried:", calls, "new call")   # 1 new call


asyncio.run(main())

Sharing failures is what protects the downstream during an outage: a hundred concurrent requests for a failing key produce one failing call per burst instead of a hundred. Not caching them is what lets the system recover as soon as the dependency does. If failures should also be suppressed for a few seconds between bursts, that is a job for a circuit breaker, not for single-flight.

Three tools against duplicate load A grid of 3 rows by 3 columns. Three tools against duplicate load tool merges remembers during an outage single-flight overlapping calls nothing one failing call per burst TTL cache calls within the TTL successful values serves until expiry circuit breaker nothing recent failures rejects calls fast They compose: breaker around the call, single-flight in front, cache in front of that.

Verify: five concurrent callers see LookupError from one call, and the next caller triggers a fresh call.

Life of one in-flight key 4 stages from first caller to key forgotten. Life of one in-flight key first caller creates task later callers join and wait task finishes result or error key forgotten next burst retries Cancelled waiters leave early; the task is cancelled only when none remain.

4. Keep one caller's cancellation from cancelling everyone

Callers get cancelled all the time — a client disconnects, a request times out. If a caller awaited the shared task directly, cancelling that caller would cancel the task, and every other waiter would receive CancelledError for work they still wanted. asyncio.shield() prevents that, and the waiter count decides when the work is genuinely unwanted.

import asyncio


async def main() -> None:
    global calls
    flight = SingleFlight()

    calls = 0
    impatient = asyncio.create_task(flight.do("u2", lambda: load_profile("u2")))
    patient = asyncio.create_task(flight.do("u2", lambda: load_profile("u2")))
    await asyncio.sleep(0.01)
    impatient.cancel()                                     # one caller gives up
    print("patient caller still gets:", await patient, "| calls:", calls)

    lonely = asyncio.create_task(flight.do("u3", lambda: load_profile("u3")))
    await asyncio.sleep(0.01)
    lonely.cancel()                                        # the only caller gives up
    await asyncio.gather(lonely, return_exceptions=True)
    await asyncio.sleep(0)
    print("in flight after the last waiter left:", list(flight._inflight))   # []


asyncio.run(main())

The patient caller receives the profile from the single call even though the other waiter was cancelled mid-flight. When the last waiter leaves, the shared task is cancelled too, so abandoned work does not keep consuming a connection. Whether to cancel at zero waiters is a policy choice: for idempotent reads that will likely be requested again, letting the call finish and populating a cache (step 5) can be the better trade.

Verify: the patient caller gets {'user': 'u2'} with one downstream call, and after the lone caller is cancelled no key remains in flight.

5. Put a short cache behind the flight

Single-flight only merges calls that overlap. A request arriving one millisecond after the shared call finished starts a new call. For hot keys, add a small time-to-live cache in front, and use single-flight to fill it, so that expiry never produces a stampede.

import asyncio
import time


class CachedFlight:
    def __init__(self, ttl: float) -> None:
        self._ttl = ttl
        self._flight = SingleFlight()
        self._values: dict[str, tuple[float, object]] = {}

    async def get(self, key: str, loader) -> object:
        hit = self._values.get(key)
        if hit is not None and hit[0] > time.monotonic():
            return hit[1]                                   # fresh: no call at all
        value = await self._flight.do(key, loader)          # stale or missing: one call per burst
        self._values[key] = (time.monotonic() + self._ttl, value)
        return value


async def main() -> None:
    global calls
    calls = 0
    cache = CachedFlight(ttl=0.2)
    for _ in range(3):                                      # three bursts, 50 ms apart
        await asyncio.gather(*(cache.get("u1", lambda: load_profile("u1")) for _ in range(100)))
        await asyncio.sleep(0.05)
    print("downstream calls for 300 requests:", calls)     # 1


asyncio.run(main())

Three hundred requests across three bursts produce a single downstream call: the first burst coalesces, and the next two are served from the cache. The full treatment of expiry, jitter and stale-while-revalidate is in preventing cache stampedes in asyncio.

Verify: the run prints one downstream call; setting ttl=0.01 makes it three, one per burst, which is still far fewer than 300.

Verification

Single-flight is working when:

  • Duplicate work collapses: downstream calls per burst equal the number of distinct keys, not the number of requests.
  • Failures are shared and forgotten: concurrent waiters see one failure, and the next caller retries.
  • Cancellation is isolated: a cancelled caller does not cancel the call for others, and the call is cancelled when nobody is left waiting.
  • Nothing lingers: the in-flight map is empty between bursts.
  • Hot keys are cached: a short TTL cache sits in front, filled through single-flight.

Pitfalls & edge cases

  • Keys that are not really identical. Two requests that differ in headers, tenant or authorisation must not share a key, or one user can receive another's result. Build keys from every input that affects the response.
  • Mutating shared results. Every waiter receives the same object. A caller that edits it changes what everyone else sees; return immutable data or copy on use.
  • Multi-process deployments. The map lives in one process. Several workers still send one call each; for cross-process coalescing use a distributed lock or accept the per-process multiplier.
  • Very long calls. A call that hangs holds every waiter hostage. Put a timeout inside fn, so the shared task itself fails, rather than timing out individual waiters.
  • Using it for writes. Coalescing two "charge the card" calls into one is a correctness bug. Single-flight is for reads and idempotent computations only.

Frequently Asked Questions

What is the single-flight pattern?

It is a way to deduplicate concurrent work: while a call for a given key is in progress, other callers requesting the same key wait for that call's result instead of starting their own. It originated as Go's singleflight package and is commonly used in front of caches and expensive downstream services to prevent thundering herds.

How do I implement single-flight in asyncio?

Keep a dictionary mapping each key to an in-flight asyncio.Task. The first caller creates the task, later callers await the same task, and a done-callback removes the key when it finishes. Because there is no await between checking and inserting the key, callers on the same event loop cannot race.

Should single-flight cache errors?

No. Concurrent callers should all receive the error from the shared call, but the key should be removed when the call fails so the next caller retries. Suppressing repeated calls to a failing dependency for a period of time is the job of a circuit breaker, not of single-flight.

What happens if one caller of a shared task is cancelled?

If callers await the shared task through asyncio.shield, cancelling one caller only cancels that caller's wait, and the other callers still receive the result. Keeping a count of waiters lets you cancel the shared task when the last interested caller has gone.