Skip to content

Avoiding Deadlocks with Nested Asyncio Locks

An async deadlock does not look like a deadlock. There is no CPU spike, no error, no stack trace — the loop keeps running, health checks keep passing, and a subset of requests simply never return. Everything looks fine except the number that matters. The two causes account for nearly all of them: two code paths acquiring the same pair of locks in opposite orders, and a coroutine that re-enters a lock it already holds, because asyncio.Lock is not reentrant and will happily wait for itself forever.

Both are avoidable by construction, and both are diagnosable in about a minute once you know that a task dump shows exactly which line every stuck task is parked on. This guide covers the reproduction, the two structural fixes (ordering and scope reduction), acquisition timeouts as a safety net, and the dump-reading technique that finds the cycle.

Prerequisites

The cycle that never resolves Four states forming a two-lock deadlock cycle. The cycle that never resolves task 1 holds A task 1 wants B task 2 holds B task 2 wants A after a yield B is taken after a yield A is taken — cycle closes One thread, two coroutines, no progress: threads were never the requirement.

1. Reproduce the classic ordering deadlock

Two locks, two orders, one hang.

import asyncio

lock_a = asyncio.Lock()
lock_b = asyncio.Lock()


async def path_one() -> None:
    async with lock_a:
        await asyncio.sleep(0)            # yield: let the other task run
        async with lock_b:                # waits for B, holding A
            ...


async def path_two() -> None:
    async with lock_b:
        await asyncio.sleep(0)
        async with lock_a:                # waits for A, holding B  -> cycle
            ...


async def main() -> None:
    await asyncio.gather(path_one(), path_two())     # never returns


asyncio.run(main())

The await asyncio.sleep(0) is what makes it deterministic; in production the yield is a database call or an HTTP request, which is why the deadlock appears only under concurrency. Note that a single-threaded event loop does not protect you here at all: the cycle is between suspended coroutines, not threads. Verify: the program hangs, and a SIGUSR1 task dump shows one task parked in lock_b.acquire() and another in lock_a.acquire().

2. Fix it with a global lock order

The standard remedy: define one order for all locks and never deviate.

import asyncio
from contextlib import asynccontextmanager

# Ordered by rank. Every path acquires in ascending rank, always.
LOCKS = {"accounts": (1, asyncio.Lock()), "ledger": (2, asyncio.Lock())}


@asynccontextmanager
async def acquire(*names: str):
    ordered = sorted(names, key=lambda n: LOCKS[n][0])      # canonical order
    acquired = []
    try:
        for name in ordered:
            await LOCKS[name][1].acquire()
            acquired.append(name)
        yield
    finally:
        for name in reversed(acquired):                     # release in reverse
            LOCKS[name][1].release()


async def transfer() -> None:
    async with acquire("ledger", "accounts"):               # order fixed for you
        ...

Sorting inside the helper is what makes the rule impossible to get wrong: call sites list the locks they need in any order, and the helper imposes the canonical one. Releasing in reverse order is conventional rather than strictly necessary for correctness, but it keeps the code symmetric and avoids surprises if you later add a lock that must be held across the release of another. Verify: run both paths from step 1 through the helper under concurrency; neither hangs, regardless of the order they list.

3. Do not re-enter a lock you already hold

asyncio.Lock is not reentrant — a coroutine that acquires it twice deadlocks against itself.

import asyncio

lock = asyncio.Lock()


# WRONG: helper re-acquires the same lock
async def update(record) -> None:
    async with lock:
        await _write(record)


async def _write(record) -> None:
    async with lock:                 # already held by this same coroutine: hangs
        ...


# RIGHT: split the locked core from the public wrapper
async def update_fixed(record) -> None:
    async with lock:
        await _write_locked(record)


async def _write_locked(record) -> None:
    """Caller MUST hold `lock`. Never acquires it itself."""
    ...

The convention that scales is naming: a _locked suffix documents that the function expects the lock to be held, so re-entry never happens by accident during refactoring. Resist the temptation to build a reentrant lock — it hides the real problem, which is that the locked region's boundaries are unclear. Verify: call the fixed version from both the wrapper and directly under the lock; neither hangs.

Re-entering a lock you already hold Two columns contrasting lock re-entry with the locked-suffix convention. Re-entering a lock you already hold helper acquires again hangs forever asyncio.Lock is not reentrant The coroutine waits on itself No error, no traceback Looks like a slow request split public and _locked correct Wrapper acquires once Internal assumes it is held Naming documents the contract Survives refactoring The convention is the fix; a reentrant lock would only hide the unclear boundary.

4. Keep awaits out of locked regions, and add a timeout

The longer a lock is held across an await, the wider the window for a cycle — and a timeout turns a hang into an error you can see.

import asyncio

lock = asyncio.Lock()


# WRONG: an HTTP call inside the critical section
async def refresh_bad(key: str) -> None:
    async with lock:
        data = await http.get(f"/values/{key}")     # 300 ms holding the lock
        cache[key] = data


# RIGHT: fetch outside, mutate inside, and bound the acquisition
async def refresh_good(key: str) -> None:
    data = await http.get(f"/values/{key}")         # no lock held
    async with asyncio.timeout(2.0):                # bound the WAIT, not the work
        await lock.acquire()
    try:
        cache[key] = data                            # no await inside
    finally:
        lock.release()

The rule of thumb is that a critical section should contain no await at all where possible: on a single-threaded loop, a block without an await is already atomic, so many "locks" turn out to be unnecessary once the remote call moves outside. Where an await is unavoidable, the acquisition timeout converts a deadlock from an invisible hang into a TimeoutError with a stack trace pointing at the exact line. Verify: with a deliberately induced cycle, the timeout fires and logs both the waiter and the lock name rather than hanging.

5. Find the cycle from a task dump

When it does hang, the dump names both ends in seconds.

import asyncio


def dump_locks() -> None:
    for task in asyncio.all_tasks():
        coro = task.get_coro()
        frame = getattr(coro, "cr_frame", None)
        if frame and "acquire" in frame.f_code.co_name:
            print(f"{task.get_name():<28} waiting at "
                  f"{frame.f_back.f_code.co_filename}:{frame.f_back.f_lineno}")

transfer#req-4a1     waiting at /srv/app/ledger.py:88     # holds accounts, wants ledger
reconcile#job-7      waiting at /srv/app/accounts.py:42   # holds ledger, wants accounts

Two tasks, each parked on a different acquire(), with the file and line of the call site — that pair is the cycle, and the fix is to make both call sites use the ordering helper from step 2. Wire this to a signal so it can be taken from a running container. Verify: trigger the dump during the step 1 hang; the two lines should name the two conflicting acquisition sites.

Four rules that remove most deadlocks Four-row matrix of lock discipline rules. Four rules that remove most deadlocks rule why rank every lock sort names in a helper call sites cannot get it wrong acquire in one order ascending rank, always a cycle needs two orders to exist no await in the section fetch outside, mutate inside shrinks the collision window to zero bound the acquisition asyncio.timeout on acquire turns a hang into a traceable error Three are structural and prevent the bug; the fourth makes it visible when one slips through.

Verification

  • No hangs under concurrency. A load test exercising every locking path concurrently completes with no stuck tasks.
  • Acquisitions are fast. Time spent waiting on any lock stays in single-digit milliseconds at peak.
  • Cycles surface as errors. A deliberately introduced ordering violation produces a TimeoutError naming the lock, not a silent stall.

Pitfalls and edge cases

  • Different acquisition orders in different modules. The most common cause; enforce order in one helper.
  • Re-entering a lock via a helper. asyncio.Lock is not reentrant; adopt a _locked naming convention.
  • Holding a lock across a network call. Multiplies the collision window and couples your latency to a remote service.
  • Locks created at import time. Older versions bound the lock to the loop at construction; build them inside async def main().
  • Mixing asyncio.Lock with threading.Lock. A thread lock blocks the whole loop; use asyncio.to_thread for code that needs the thread lock.
  • Assuming single-threaded means safe. Deadlocks between suspended coroutines are unaffected by having one thread.
  • Locks where none are needed. A block containing no await cannot be interrupted on a single loop; the lock adds risk with no benefit.

Frequently Asked Questions

Can asyncio code deadlock on a single thread?

Yes. A deadlock is a cycle of waiting, not a property of threads. Two coroutines that acquire the same two locks in opposite orders will each hold one and wait for the other forever, and the loop keeps running happily around them — which is why the symptom is a subset of requests that never return rather than a crash or a CPU spike.

Is asyncio.Lock reentrant?

No. A coroutine that already holds an asyncio.Lock and tries to acquire it again waits for itself indefinitely. Split the code into a public function that acquires the lock and an internal one — conventionally suffixed _locked — that assumes it is already held, rather than building a reentrant lock that hides unclear critical-section boundaries.

How do I prevent lock-ordering deadlocks?

Give every lock a rank and acquire in ascending rank, always. Implement it once in a helper that sorts the requested locks before acquiring, so call sites can list them in any order and still get the canonical sequence. Release in reverse order to keep the structure symmetric.

How do I find where an asyncio deadlock is?

Take a task dump and look for tasks whose current frame is inside an acquire() call. Printing the caller's file and line for each gives you the two (or more) call sites forming the cycle. Wire the dump to a signal handler so it can be triggered on a running process during the hang.