Skip to content

Async Comprehensions and Their Costs

Python has two kinds of asynchronous comprehension, and both are easy to misread. [x async for x in source] consumes an async iterator; [await f(x) for x in xs] awaits inside an ordinary comprehension. Neither runs anything concurrently, and the first is often used where no asynchrony is needed. Measured on Python 3.14 over a million integers: a list comprehension over a sync generator took 43 ns per item; over an async generator, 154 ns; an explicit async for loop with append, 136 ns; an async iterator class, 118 ns. Consuming that million-item async generator, which never awaited anything that suspends, blocked the event loop for 129 ms — an async for is not a yield point unless the iterator actually waits. And [await fetch(i) for i in range(100)], where each call sleeps 10 ms, took 1,032 ms: the calls ran one after another. asyncio.gather over the same calls took 12 ms, and a comprehension over asyncio.as_completed 14 ms. This guide shows what each form costs and when to use something else.

Prerequisites

1. Know the two forms and what they do

The async for form iterates an async iterator and collects its items; the await form calls coroutines inside an ordinary loop and waits for each in turn:

async def main():
    rows = [row async for row in cursor]                    # consume an async iterator
    users = [await fetch_user(uid) for uid in user_ids]     # sequential awaits
    evens = {x async for x in numbers() if x % 2 == 0}      # set form, with a filter
    lazy = (x * 2 async for x in numbers())                 # async generator expression

Both are allowed only inside async def; elsewhere the compiler reports "asynchronous comprehension outside of an asynchronous function". The parenthesised form creates an async_generator, not a value — passing it to sum() raised "'async_generator' object is not iterable", because built-ins like sum, max and sorted only accept sync iterables. Collect it first with a list comprehension, or write the aggregation as an async for loop. Since Python 3.11, an async comprehension can also appear inside a sync comprehension in an async function, for example [[x async for x in rows(i)] for i in range(3)].

Verify: each comprehension in your async code is either consuming a genuinely asynchronous source or deliberately awaiting in sequence.

2. Do not use async generators for synchronous data

An async generator's per-item cost is several times a sync generator's, because each item goes through the coroutine machinery — __anext__ returns an awaitable that must be driven to completion:

def numbers_sync(n):
    for i in range(n):
        yield i

async def numbers_async(n):          # same data, nothing awaited
    for i in range(n):
        yield i

Measured over 1,000,000 items: [x for x in numbers_sync(n)] 43.4 ms, [x async for x in numbers_async(n)] 153.7 ms, an async for loop with append 136.2 ms, and a class with __anext__ 118.4 ms. If the data source never waits — an in-memory list, a parsed file already loaded, a computed range — make it a plain generator and use a plain comprehension, even inside async code. Reserve async iteration for sources that await between items, such as database cursors, network streams and paginated APIs. When wrapping a blocking source, batch items per thread call as described in adapting blocking iterators to async iterators.

Verify: async generators in your code contain at least one await that can actually suspend.

Nanoseconds per item, 1,000,000 integers 5 horizontal bars comparing sum(sync generator) with the others. Nanoseconds per item, 1,000,000 integers sum(sync generator) 18 ns [x for x in sync gen] 43 ns [x async for x in __anext__ class] 118 ns async for + append, async gen 136 ns [x async for x in async gen] 154 ns Python 3.14; none of the async sources awaited anything. Async iteration costs about 3x per item, for nothing if the source never waits.

3. Remember that async for is not a yield point

An event loop can only run other tasks when the current one suspends. Iterating an async generator suspends only if the generator awaits something that is not yet ready; one that just produces values runs straight through:

async def all_items():
    return [x async for x in numbers_async(1_000_000)]     # never suspends

Measured with a task sampling loop lag every millisecond: consuming the million-item generator blocked the loop for 129 ms, exactly as a synchronous loop would. Adding await asyncio.sleep(0) every 1,000 items kept loop lag at 0.7 ms and the total at 148 ms. Long-running iteration over fast in-memory sources needs explicit yield points, or belongs in a worker thread altogether — the question of how often to yield is the same one discussed in implementing cooperative cancellation in CPU loops.

Verify: loop lag measured while your largest async comprehension runs stays within budget.

4. Use gather or as_completed when calls are independent

[await f(x) for x in xs] starts each call only after the previous one finished. For independent calls, that is the slowest possible order:

users = [await fetch_user(uid) for uid in ids]                       # sequential

users = await asyncio.gather(*(fetch_user(uid) for uid in ids))       # concurrent, in order

async with asyncio.TaskGroup() as tg:                                 # concurrent, fail fast
    tasks = [tg.create_task(fetch_user(uid)) for uid in ids]
users = [t.result() for t in tasks]

Measured with 100 calls of 10 ms each: the comprehension took 1,032 ms; gather 12 ms; a comprehension over asyncio.as_completed, which yields results as they finish rather than in input order, 14 ms. Sequential awaiting is correct when each call depends on the previous one or when the downstream cannot take concurrent requests; otherwise it is an accidental for loop with a network round trip per element. For large inputs, bound the concurrency as in limiting concurrent requests with asyncio.Semaphore rather than gathering everything at once.

Verify: comprehensions with await are sequential on purpose, and independent calls use gather, a TaskGroup or as_completed.

100 independent calls of 10 ms each A grid of 3 rows by 4 columns. 100 independent calls of 10 ms each form time order of results concurrency [await fetch(i) for i in ids] 1,032 ms input order none await asyncio.gather(*(fetch(i) for i in ids)) 12 ms input order all [await c for c in asyncio.as_completed(...)] 14 ms completion order all Python 3.14; each call is await asyncio.sleep(0.01).

5. Stream instead of collecting when the source is large

An async list comprehension materialises everything before returning. For a large or unbounded source, that means unbounded memory and no work until the last item arrives. Process items as they come, and keep the generator closable:

from contextlib import aclosing

async def export(cursor_rows, out):
    async with aclosing(cursor_rows) as rows:
        async for row in rows:
            if row["active"]:
                await out.write(serialise(row))

aclosing guarantees the generator's cleanup runs when the loop exits early — on break, an exception or cancellation — instead of whenever the garbage collector gets to it, as described in closing async generators with aclosing. Comprehensions cannot break, so they always consume the whole source; use a loop whenever you might stop early. A filtering async comprehension over a database cursor still fetches every row the query returns — push filters into the query instead.

Verify: comprehensions are only used over sources small enough to hold in memory, and early exits use a loop with aclosing.

Which form fits this iteration? A decision on What are you iterating with 4 outcomes. Which form fits this iteration? What are you iterating? data that never waits sync comprehension 43 ns vs 154 ns per item small async source [x async for x in src] collects everything large / unbounded async source async for + aclosing streams, can stop independent coroutine calls gather / TaskGroup 12 ms vs 1,032 ms Sequential [await f(x) ...] only when each call depends on the last.

Verification

Async comprehensions are used well when:

  • Async generators exist only for sources that wait, and synchronous data uses plain generators.
  • Long iterations over fast sources yield or run in a thread.
  • Independent calls run concurrently, and await inside a comprehension is deliberate.
  • Large sources are streamed with async for and aclosing, not collected.

Diagnostic Hook: when loop lag spikes without any CPU-heavy code in the profile, look for async comprehensions over fast sources. They appear in profiles as time in async_generator_asend or __anext__ frames, and they block the loop just as a synchronous loop would.

Pitfalls & edge cases

  • Async generators over in-memory data. Measured: 154 ns per item against 43.
  • Assuming async for yields to other tasks. Measured: 129 ms of loop lag.
  • [await f(x) for x in xs] for independent calls. Measured: 1,032 ms against 12 ms.
  • Passing an async generator expression to sum or sorted. It is not iterable.

Frequently Asked Questions

Do async comprehensions run concurrently?

No. [x async for x in src] consumes items one at a time, and [await f(x) for x in xs] awaits each call in turn: 100 calls of 10 ms took 1,032 ms, against 12 ms with asyncio.gather.

Are async generators slower than normal generators?

Yes: 118 to 154 ns per item against 43 ns for a list comprehension over a sync generator in testing. Use them only when the source actually awaits.

Does iterating an async generator let other tasks run?

Only when the generator awaits something not yet ready. A million-item async generator that never waited blocked the loop for 129 ms.

Why can't I call sum() on an async generator expression?

Built-ins accept only sync iterables; sum((x async for x in src)) raised "'async_generator' object is not iterable". Collect it with a list comprehension or loop with async for.