Skip to content

Querying DynamoDB with aioboto3

DynamoDB is accessed entirely through HTTP calls, which makes it a natural fit for asyncio: queries for different partition keys are independent and can run concurrently. The limits are the client's own CPU — every item is deserialised from DynamoDB's typed JSON in Python — and the usual race conditions of concurrent writers. Measured with aioboto3 15.5 against DynamoDB Local through a proxy adding 10 ms per request, on a table of 20,000 orders across 200 customers: querying customers one after another took 0.85–1.07 s per 5,000 items; sixteen concurrent queries fetched all 20,000 in 0.35–0.52 s, with client CPU time — 38 µs per item with the low-level client, 46 µs with the resource API — nearly equal to wall time. Two hundred get_item calls took 2.41 s sequentially and 0.16 s sixteen at a time; batch_get_item fetched them in 2 calls and 0.05 s. Fifty concurrent increments of one counter by read-modify-write left it at 2; with a conditional write and retry it reached 50 after 1,014 retries; with an atomic ADD it reached 50 with none. This guide covers reading and writing DynamoDB from async code.

Prerequisites

1. Choose between the client and the resource API

aioboto3 offers both of boto3's interfaces. The resource API converts DynamoDB's typed values ({"N": "218.34"}) into Python values for you; the client returns them raw. The conversion costs CPU, and it returns numbers as Decimal:

from boto3.dynamodb.conditions import Key

async with session.resource("dynamodb") as ddb:
    table = await ddb.Table("orders")
    resp = await table.query(KeyConditionExpression=Key("customer").eq("c0001"))
    item = resp["Items"][0]
    item["total"]            # Decimal('218.34'), not float

Measured over the same queries: 46 µs of client CPU per item with the resource API, 38 µs with the client and its paginator — both far more than the network cost per item, since a query returns many items per round trip. json.dumps of a resource item raised "Object of type Decimal is not JSON serializable"; serialise with a default= that converts Decimal, or convert at the boundary with the type you mean. Use the resource API where code clarity matters, and the client with its own minimal conversion on hot paths that move many items.

Verify: you know the per-item CPU cost of your access layer, and API responses do not fail on Decimal.

2. Query partitions concurrently, within the client's CPU

Different partition keys are independent queries. Run them concurrently with a bound, and paginate each one — a query returns at most 1 MB per call:

async def orders_for(client, customer: str) -> list[dict]:
    items = []
    paginator = client.get_paginator("query")
    async for page in paginator.paginate(
        TableName="orders",
        KeyConditionExpression="customer = :c",
        ExpressionAttributeValues={":c": {"S": customer}},
    ):
        items.extend(page["Items"])
    return items

async def orders_for_many(client, customers: list[str], limit: int = 16):
    sem = asyncio.Semaphore(limit)
    async def one(c):
        async with sem:
            return await orders_for(client, c)
    return await asyncio.gather(*(one(c) for c in customers))

Measured with 10 ms per request: 50 customers sequentially, 0.85 s with the client and 1.07 s with the resource API; all 200 customers sixteen at a time, 0.35 s and 0.52 s. At that point client CPU was 0.34 s and 0.49 s — the process was spending almost all its time deserialising, and more concurrency would only add loop lag. Size max_pool_connections to at least the concurrency limit; the default of 10 would silently cap 16 concurrent queries at 10.

Verify: concurrent queries return the same items as sequential ones, and client CPU time is below wall time.

Reading from DynamoDB Local with 10 ms per request A grid of 6 rows by 4 columns. Reading from DynamoDB Local with 10 ms per request operation items wall client CPU query, 50 customers, sequential (client) 5,000 0.85 s 0.19 s query, 200 customers, 16 concurrent (client) 20,000 0.35 s 0.34 s query, 200 customers, 16 concurrent (resource) 20,000 0.52 s 0.49 s get_item x200, sequential 200 2.41 s - get_item x200, 16 concurrent 200 0.16 s - batch_get_item, 2 calls of 100 200 0.05 s - aioboto3 15.5; deserialisation cost 38-46 us of CPU per item.

3. Batch point reads, and retry what comes back unprocessed

For many known keys, batch_get_item fetches up to 100 per call. It may return some keys under UnprocessedKeys when throughput is constrained, and those must be requested again — with backoff, because the cause is throttling:

async def batch_get(client, table: str, keys: list[dict]) -> list[dict]:
    items: list[dict] = []
    for i in range(0, len(keys), 100):
        request = {table: {"Keys": keys[i:i + 100]}}
        attempt = 0
        while request:
            resp = await client.batch_get_item(RequestItems=request)
            items.extend(resp["Responses"].get(table, []))
            request = resp.get("UnprocessedKeys") or None
            if request:
                attempt += 1
                await asyncio.sleep(min(2.0, 0.05 * 2 ** attempt))
    return items

Measured for 200 keys: two calls and 0.05 s, against 200 calls and 0.16 s with sixteen concurrent get_item calls, and 2.41 s one at a time. DynamoDB Local never throttles, so the UnprocessedKeys loop did not run here — test it against a mock that returns unprocessed keys, because code paths that never run in development are the ones that fail in production. The same applies to batch_write_item and UnprocessedItems; the resource API's batch_writer() handles that retry for you, which is a good reason to use it for bulk writes.

Verify: a test with a fake response containing UnprocessedKeys shows the keys being retried until none remain.

4. Never read-modify-write without a condition

Concurrent writers that read an item, change it and put it back overwrite each other. DynamoDB has two ways to prevent that: an atomic update expression, when the change can be expressed as one, and a conditional write with retry, when it cannot:

# atomic: the server applies the change
await client.update_item(
    TableName="orders", Key=key,
    UpdateExpression="ADD n :one", ExpressionAttributeValues={":one": {"N": "1"}},
)

# optimistic: write only if nobody else wrote since we read
async def update_with_version(client, key, change) -> None:
    while True:
        item = (await client.get_item(TableName="orders", Key=key, ConsistentRead=True))["Item"]
        version = int(item["version"]["N"])
        new = change(item) | {"version": {"N": str(version + 1)}}
        try:
            await client.put_item(
                TableName="orders", Item=new,
                ConditionExpression="version = :v",
                ExpressionAttributeValues={":v": {"N": str(version)}},
            )
            return
        except ClientError as exc:
            if exc.response["Error"]["Code"] != "ConditionalCheckFailedException":
                raise

Measured with 50 concurrent increments of one item: plain read-modify-write ended at n = 2 — 48 updates lost, silently. The conditional version ended at 50 but needed 1,014 retries and 1.51 s, because every writer contended for the same item. The atomic ADD ended at 50 with no retries in 0.06 s. Prefer update expressions (ADD, SET x = x + :d, list_append, conditional SET) wherever the change fits; use versioned conditional writes for changes that need the old value in application code, and add jittered backoff between retries when contention is high.

Verify: a concurrent-update test of your write path ends with the exact expected value.

50 concurrent increments of one item A grid of 3 rows by 4 columns. 50 concurrent increments of one item strategy final value retries time get_item, then put_item 2 0 0.13 s versioned put_item + ConditionExpression 50 1,014 1.51 s update_item ADD n :one 50 0 0.06 s The wrong version was the fastest and silently lost 96% of updates.

5. Scan in parallel segments, sparingly

A Scan reads the whole table; parallel scan splits it into segments that can be read concurrently:

async def scan_all(client, table: str, segments: int = 4):
    async def segment(i):
        items = []
        async for page in client.get_paginator("scan").paginate(
                TableName=table, Segment=i, TotalSegments=segments):
            items.extend(page["Items"])
        return items
    parts = await asyncio.gather(*(segment(i) for i in range(segments)))
    return [item for part in parts for item in part]

Measured on 20,000 items: one segment 0.83 s, four 0.50 s, eight 0.43 s, with client CPU at about 0.4 s in every case — again the deserialisation, not the network, set the floor. Scans consume read capacity for every item examined, filters or not, so they belong in batch jobs, not request handlers; for repeated whole-table reads, an export to S3 avoids consuming table capacity. If a scan must run alongside live traffic, keep the segment count low and throttle it, as in Rate Limiting & Throttling.

Verify: scans run outside request paths, and their segment count is chosen against the table's capacity, not the client's speed.

Which DynamoDB operation for this access? A decision on What do you need to read or write with 4 outcomes. Which DynamoDB operation for this access? What do you need to read or write? one partition's items paginated query 1 MB per page many partitions concurrent queries, bounded 3x faster, then CPU-bound many known keys batch_get_item + UnprocessedKeys 2 calls for 200 a counter or field update update expression, else versioned condition 0 lost Scans are for batch jobs only.

Verification

DynamoDB access from asyncio is sound when:

  • Per-item deserialisation cost is known, and Decimal is handled at the boundary.
  • Concurrent queries are bounded, with a pool at least as large as the bound.
  • Batch operations retry unprocessed keys with backoff, and that path is tested.
  • Every concurrent update is atomic or conditional, proven by a test that counts.

Diagnostic Hook: log consumed capacity (ReturnConsumedCapacity="TOTAL") per operation type and count ConditionalCheckFailedExceptions. A rising conditional-failure rate means writers are contending for the same items — a hot key — and each failure is a full read and write repeated; the fix is usually an update expression or a different key design, not more retries.

Pitfalls & edge cases

  • Read-modify-write. Measured: 48 of 50 updates lost, without an error.
  • Assuming DynamoDB calls are network-bound. Deserialisation took 38–46 µs per item.
  • The default pool of 10. It silently caps concurrency.
  • Untested UnprocessedKeys handling. DynamoDB Local never returns any.

Frequently Asked Questions

How do I query DynamoDB asynchronously in Python?

Use aioboto3's client or resource with async with, paginate each query, and run independent partition-key queries concurrently behind a semaphore. Sixteen concurrent queries fetched 20,000 items in 0.35 s against 0.85 s per 5,000 sequentially in testing.

Should I use the aioboto3 resource or client for DynamoDB?

The resource converts types for you but returns Decimal and cost 46 µs of CPU per item against 38 µs for the client. Use the client on hot paths.

How do I prevent lost updates in DynamoDB?

Use an atomic update expression such as ADD, or a conditional write on a version attribute with retry. Plain read-modify-write left a counter at 2 after 50 concurrent increments.

Is batch_get_item faster than many get_item calls?

Yes: 200 items took 2 calls and 0.05 s, against 0.16 s with 16 concurrent get_item calls. Always retry UnprocessedKeys.