Skip to content

Idempotency Keys for Safe Async Retries

The payment service timed out after 5 seconds, the client retried, and the customer was charged twice. Nothing in the retry loop was wrong: the backoff was jittered, the attempt cap was sensible, the timeout was reasonable. The problem is that a timeout does not tell you whether the work happened. The server may have committed the charge and lost the reply on the way back, and to the client that looks exactly like a request that never arrived. Retrying a non-idempotent operation under that uncertainty is a coin toss between "lost payment" and "double payment". Idempotency keys remove the uncertainty: the client names each logical operation once, sends the same name on every retry, and the server guarantees that one name produces at most one side effect. This guide builds both halves in asyncio, including the part most implementations get wrong — concurrent duplicates arriving while the first attempt is still running.

Prerequisites

  • Python 3.11+, standard library only; every snippet runs under asyncio.run().
  • Retry fundamentals from Retry & Backoff Strategies and exponential backoff with jitter. This page makes those retries safe; it does not replace them.
  • Timeout semantics from Timeouts & Deadlines: a timeout on the client says nothing about completion on the server.
  • Control over the server, or an upstream that already accepts an Idempotency-Key header (most payment and messaging APIs do).
A lost response makes the outcome unknown 3 lanes over time. A lost response makes the outcome unknown client send waiting for reply timeout retry waiting server charge committed reply lost same key: stored result without key second charge time → The client cannot tell a lost reply from a lost request; the key lets the server tell.

1. Generate one key per logical operation

The key identifies the thing the user wants to happen once, not the HTTP request. Generate it where that intent is created — when the order is submitted, when the job is enqueued — and carry it through every retry. A key generated inside the retry loop is a new key per attempt and protects nothing.

import uuid
from dataclasses import dataclass, field


@dataclass(frozen=True)
class ChargeRequest:
    account: str
    amount_cents: int
    # Created once with the intent, reused by every attempt and every process
    # that handles this request (persist it with the order if retries can span restarts).
    idempotency_key: str = field(default_factory=lambda: str(uuid.uuid4()))


def headers_for(req: ChargeRequest) -> dict[str, str]:
    return {"Idempotency-Key": req.idempotency_key}

If a retry can happen after a process restart — a job re-delivered by a queue, a user pressing "pay" again after a crash — the key must be stored with the job or the order row, not only held in memory. A deterministic key derived from business identity (f"order-{order_id}-charge") is often better than a random one for exactly that reason.

Verify: log the key alongside the attempt number. Every attempt for one order carries the same key; two different orders never share one.

2. Retry with the same key on every attempt

The client's retry loop becomes simple once the key exists: every failure that leaves the outcome unknown — a timeout, a reset connection, a 502 from a proxy — is retried with the identical key and payload. Failures that are definitely final, such as a 400 validation error, are not retried at all.

import asyncio
import random

UNKNOWN_OUTCOME = (TimeoutError, ConnectionError)


async def charge_with_retries(send, req: ChargeRequest, attempts: int = 4) -> dict:
    payload = {"account": req.account, "amount_cents": req.amount_cents}
    for attempt in range(1, attempts + 1):
        try:
            async with asyncio.timeout(5.0):
                return await send(payload, headers_for(req))   # same key every time
        except UNKNOWN_OUTCOME:
            if attempt == attempts:
                raise
            await asyncio.sleep(random.uniform(0, 0.2 * 2 ** attempt))
    raise AssertionError("unreachable")

Verify: inject a failure that drops the response after the server has done the work. The client retries, and the second response carries the result of the first execution rather than a new side effect — which requires the server side below.

3. Claim the key before any side effect

On the server, the guarantee comes from ordering: the key must be recorded as in progress before the charge happens, atomically, so two requests with the same key cannot both pass the check. In-process this is a dictionary write with no await between the lookup and the insert; across processes it is a unique constraint.

import asyncio
import hashlib
import json
from dataclasses import dataclass


class IdempotencyConflict(Exception):
    """Same key, different request body."""


@dataclass
class Entry:
    fingerprint: str
    outcome: asyncio.Future


class IdempotencyStore:
    def __init__(self) -> None:
        self._entries: dict[str, Entry] = {}

    @staticmethod
    def fingerprint(payload: dict) -> str:
        return hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()

    async def run_once(self, key: str, payload: dict, operation) -> dict:
        fp = self.fingerprint(payload)
        entry = self._entries.get(key)
        if entry is not None:                         # seen before: see step 4
            if entry.fingerprint != fp:
                raise IdempotencyConflict(f"key {key!r} reused with a different payload")
            return await asyncio.shield(entry.outcome)
        entry = Entry(fp, asyncio.get_running_loop().create_future())
        self._entries[key] = entry                    # claimed: no await since the lookup
        try:
            result = await operation(payload)
        except asyncio.CancelledError:
            del self._entries[key]
            entry.outcome.cancel()
            raise
        except Exception as exc:
            del self._entries[key]                    # nothing committed: a retry may run it
            entry.outcome.set_exception(exc)
            entry.outcome.exception()                 # mark retrieved to silence the warning
            raise
        entry.outcome.set_result(result)
        return result

The absence of an await between self._entries.get(key) and self._entries[key] = entry is what makes this atomic on a single event loop: no other coroutine can run in between. The moment the store moves to Postgres or Redis, that guarantee must come from the database instead — an INSERT ... ON CONFLICT DO NOTHING on the key column, or SET key value NX — because several processes are now racing.

Verify: run the lost-response scenario from step 2 against this store. The ledger records exactly one charge, and the retry returns the same result object as the first execution.

Lifecycle of one idempotency key 4 stages from absent to expired. Lifecycle of one idempotency key absent first request claim in progress duplicates wait commit completed result replayed TTL expired after the TTL A failure before commit returns the key to absent so the next retry can run.

4. Coalesce concurrent duplicates

Retries are not always sequential. A client with an aggressive timeout can send attempt two while attempt one is still executing, and a user can double-click. The store above handles this by keeping the outcome as a Future: a duplicate that arrives mid-flight awaits the same future instead of starting a second execution or failing.

import asyncio
import uuid

charges: list[int] = []


async def charge(payload: dict) -> dict:
    await asyncio.sleep(0.05)                          # the slow, non-idempotent part
    charges.append(payload["amount_cents"])
    return {"status": "charged", "amount_cents": payload["amount_cents"]}


async def main() -> None:
    store = IdempotencyStore()
    key = str(uuid.uuid4())
    payload = {"account": "acct-7", "amount_cents": 500}
    results = await asyncio.gather(*(store.run_once(key, payload, charge) for _ in range(5)))
    print(len(charges), all(r == results[0] for r in results))   # 1 True


asyncio.run(main())

asyncio.shield() matters here. If a duplicate caller is cancelled — its client disconnected — the shield stops that cancellation from propagating into the shared future and cancelling the one execution everyone else is waiting on. The pattern is covered in depth in using asyncio.shield to protect critical sections.

Verify: five concurrent calls with one key produce one charge and five identical results. Cancel one of the duplicate callers mid-flight; the others still receive the result.

5. Reject mismatched payloads and expire old keys

Two rules keep the store honest over time. A key reused with a different payload is a client bug, and returning the stored result would silently hide it — reject it instead (HTTP APIs typically use 422). And keys cannot live forever: keep completed entries long enough to cover the longest realistic retry window, then expire them.

import asyncio
import time


class ExpiringIdempotencyStore(IdempotencyStore):
    def __init__(self, ttl_seconds: float = 24 * 3600) -> None:
        super().__init__()
        self._ttl = ttl_seconds
        self._completed_at: dict[str, float] = {}

    async def run_once(self, key: str, payload: dict, operation) -> dict:
        result = await super().run_once(key, payload, operation)
        self._completed_at.setdefault(key, time.monotonic())
        return result

    async def sweep_forever(self, interval: float = 60.0) -> None:
        while True:
            await asyncio.sleep(interval)
            cutoff = time.monotonic() - self._ttl
            for key, finished in list(self._completed_at.items()):
                if finished < cutoff:
                    self._entries.pop(key, None)
                    del self._completed_at[key]

Choose the TTL from the retry policy, not from storage cost: if a queue can redeliver a message for up to three days, a 24-hour key TTL reopens the double-execution window on day two. Run the sweeper as a supervised background task and keep a strong reference to it.

Verify: a replay with the same key and a changed amount raises IdempotencyConflict; a replay after the TTL executes again, which is the documented behaviour, and store size stays flat under steady load.

What the server does with a request A grid of 4 rows by 2 columns. What the server does with a request key state same payload different payload not seen claim and execute claim and execute in progress await shared outcome reject (422) completed return stored result reject (422) expired execute again execute again Only the first row ever performs the side effect while the key is retained.

Verification

Idempotent retries are working when:

  • Lost responses do not duplicate work: fault injection that drops replies after commit yields exactly one side effect per logical operation.
  • Concurrent duplicates coalesce: simultaneous requests with one key execute once and all receive the same result.
  • Failures before commit are retryable: an operation that raised before its side effect releases the key, and the next attempt runs normally.
  • Misuse is loud: a key reused with a different payload is rejected rather than answered with a stale result.
  • Keys outlive retries: the TTL exceeds the longest redelivery or retry window, and the store's size is bounded by that TTL.

Pitfalls & edge cases

  • Releasing the key after a partial commit. If the operation writes to the database and then fails while publishing an event, deleting the key allows a second write. Record completion in the same transaction as the side effect, or make each sub-step idempotent in its own right.
  • In-memory stores behind a load balancer. Two instances each hold their own dictionary, so a retry routed to the other instance executes again. Anything beyond a single process needs a shared store with an atomic insert.
  • Keys generated per attempt. The most common bug: uuid4() inside the retry loop. The key must be created with the intent and passed in.
  • Caching error responses forever. Storing a transient 503 as the outcome makes every retry return the same failure. Only store outcomes that are final — success or a definitive validation error.
  • Timeouts shorter than execution. A duplicate that waits on the shared future inherits the first attempt's duration. Give duplicates a wait bound with asyncio.timeout() so a stuck execution does not pin every retry.

Frequently Asked Questions

What is an idempotency key?

An idempotency key is a unique identifier the client attaches to one logical operation and repeats on every retry of it. The server records the key before performing the side effect and returns the stored result for any later request with the same key, so retries after timeouts or lost responses cannot perform the operation twice.

Where should the idempotency key be generated?

Where the intent is created, such as when an order is submitted or a job is enqueued, and then carried through every attempt. Generating a new key inside the retry loop defeats the purpose. If retries can span process restarts or queue redeliveries, persist the key with the job, or derive it deterministically from business identifiers.

How do I handle two requests with the same idempotency key at the same time?

Record the key as in progress with a shared future or database row before doing the work, and make duplicates wait for that outcome instead of executing again. In a single asyncio process the lookup and insert must happen with no await in between; across processes use an atomic insert such as a unique constraint or SET NX.

How long should idempotency keys be kept?

Longer than the longest window in which a retry or redelivery can arrive. If a message queue can redeliver for three days, keys must survive at least that long. Many public APIs keep keys for about 24 hours because their clients retry within minutes; derive the value from your own retry policy.