Skip to content

Generating Presigned URLs with aioboto3

A presigned URL hands a client temporary permission for one S3 operation, so uploads and downloads go directly between the client and the bucket instead of through your async service. Generating one is local computation — no request to S3 — which makes it easy to get wrong in ways that only show up later. Measured with aioboto3 15.5 against SeaweedFS, an S3-compatible server that validates signatures, conditions and expiry: signing took 189–222 µs per URL with a shared client and 2.6–3.6 ms with a client created per URL. Because await client.generate_presigned_url(...) never actually suspends, signing 10,000 URLs in one request stalled the event loop for the full 1.9–2.2 s; yielding every 100 URLs brought the stall to 59 ms, and signing in a thread with boto3 to 10 ms. A presigned PUT that included ContentType rejected an upload with a different Content-Type with 403 SignatureDoesNotMatch; a URL that had expired returned 403 Request has expired; a presigned POST with a content-length-range condition of 10 KB accepted 5 KB and rejected 50 KB with 400 EntityTooLarge; SigV4 URLs longer than a week were refused with 400. The same checks against moto, a popular local AWS mock, all passed — it validates none of them. This guide covers generating presigned URLs correctly from async code.

Prerequisites

1. Sign with SigV4 and a shared client

Create one S3 client at start-up and reuse it, and request Signature Version 4 explicitly — with a custom endpoint and no signature_version set, botocore produced legacy SigV2 URLs (AWSAccessKeyId=...&Signature=...) in testing:

import aioboto3
from botocore.config import Config

session = aioboto3.Session()

async def lifespan(app):
    async with session.client("s3", config=Config(signature_version="s3v4")) as s3:
        app.state.s3 = s3
        yield

async def download_link(s3, key: str) -> str:
    return await s3.generate_presigned_url(
        "get_object",
        Params={"Bucket": "reports", "Key": key},
        ExpiresIn=300,
    )

Measured: 189 µs per URL with SigV2 and 222 µs with SigV4 on a shared client; 2.6–3.6 ms per URL when each one created its own client. Signing does not contact S3 — a URL was produced against an endpoint with nothing listening — so it cannot tell you whether the key exists or the credentials are allowed to read it; those failures happen when the client uses the URL. Check authorisation in your own handler before signing: a presigned URL grants whatever the signing credentials can do for that operation and key.

Verify: generated URLs contain X-Amz-Algorithm=AWS4-HMAC-SHA256, and one client is shared across requests.

2. Do not sign thousands of URLs on the loop

generate_presigned_url in aiobotocore is a coroutine, but it does no I/O, so awaiting it never yields to the event loop. A handler that signs a URL for every item in a large listing blocks the loop for the whole batch:

async def sign_listing(s3, keys: list[str]) -> list[str]:
    urls = []
    for i, key in enumerate(keys):
        urls.append(await s3.generate_presigned_url(
            "get_object", Params={"Bucket": "reports", "Key": key}, ExpiresIn=300))
        if i % 100 == 99:
            await asyncio.sleep(0)            # let other requests run
    return urls

Measured with 10,000 keys: without the sleep(0), the maximum loop lag equalled the whole run, 1.9–2.2 s. Yielding every 100 URLs kept the total at 1.82 s and cut the lag to 59 ms. Signing the batch in a worker thread with a synchronous boto3 client took 1.76 s with 10.2 ms of lag. Better still is not signing what the client will not use: page the listing and sign only the visible page, or sign on demand when the user clicks.

Verify: loop lag during your largest signing request stays within budget.

Signing 10,000 presigned GET URLs A grid of 4 rows by 4 columns. Signing 10,000 presigned GET URLs approach per URL 10,000 URLs max loop lag await in a loop, shared client 189-222 us 1.89-2.22 s the whole run await, sleep(0) every 100 182 us 1.82 s 59 ms sync boto3 in to_thread 176 us 1.76 s 10.2 ms new client per URL 2.6-3.6 ms - 264-359 ms per 100 aioboto3 15.5; signing never suspends, so it never yields.

3. Constrain uploads with signed headers or POST policies

A presigned PUT signs the parameters you pass, so include everything the upload must match. ContentType becomes part of the signature, and the client must send exactly that header:

async def upload_link(s3, user_id: str) -> str:
    key = f"avatars/{user_id}.png"
    return await s3.generate_presigned_url(
        "put_object",
        Params={"Bucket": "uploads", "Key": key, "ContentType": "image/png"},
        ExpiresIn=60,
    )

Measured: an upload with Content-Type: image/png returned 200; the same URL with Content-Type: text/html returned 403 SignatureDoesNotMatch — so a client cannot use an image upload URL to plant an HTML file. A PUT URL cannot limit size. For that, use a presigned POST, whose policy can carry conditions:

post = await s3.generate_presigned_post(
    "uploads",
    "avatars/${filename}",
    Fields={"Content-Type": "image/png"},
    Conditions=[["content-length-range", 1, 10_000], {"Content-Type": "image/png"}],
    ExpiresIn=60,
)
# hand post["url"] and post["fields"] to the browser, which submits a multipart form

Measured: a 5 KB file returned 204 and was stored; a 50 KB file returned 400 EntityTooLarge and was not. Choose the key on the server, not from the client's filename, unless the policy restricts it — ${filename} here is for illustration and should be a fixed prefix plus an ID in production.

Verify: an upload with the wrong content type or above the size limit is rejected by the storage service, not by your code.

What the storage service enforced A grid of 5 rows by 3 columns. What the storage service enforced request SeaweedFS (validates) moto (mock) PUT, signed Content-Type 200 200 PUT, other Content-Type 403 SignatureDoesNotMatch 200 GET after ExpiresIn 403 Request has expired 200 POST 50 KB, range 1-10,000 400 EntityTooLarge 204 SigV4, ExpiresIn 604,801 s 400 must be less than a week - Test presigned URLs against a server that checks them.

4. Keep expiry short and bounded by the credentials

A presigned URL is a bearer token: anyone holding it can use it until it expires. Measured: a URL with ExpiresIn=2 returned 200 immediately and 403 "Request has expired" three seconds later. SigV4 allows at most seven days — 604,800 s worked, 604,801 s was rejected with 400 "X-Amz-Expires must be less than a week". Pick the shortest expiry that covers the client's use:

DOWNLOAD_TTL = 300        # long enough to start a download, short enough to leak harmlessly
UPLOAD_TTL = 60           # the browser uploads immediately after receiving it

The URL also stops working when the credentials that signed it stop working. A service running with temporary credentials — an IAM role on ECS, Lambda or EKS — signs with a session token, and its URLs die when that session expires, which can be sooner than ExpiresIn. For links that must outlive the signing process's credentials, sign with longer-lived credentials or re-sign on demand. Never log full presigned URLs: the query string is the credential.

Verify: an expired URL is rejected, and logs show the key and expiry, not the signed query string.

5. Test against a server that validates signatures

The checks above are only as good as the server they ran against. The same suite run against moto — whose S3 mock accepts presigned requests without checking them — returned 200 for the wrong content type, 200 after expiry and 204 for the oversized upload. Tests that pass against such a mock prove that your code produces URLs, not that the URLs are restricted. Use an S3-compatible server that enforces SigV4 in integration tests, and keep moto for logic that does not depend on authorisation:

@pytest.fixture(scope="session")
def s3_endpoint():
    # an S3-compatible server that validates SigV4, started by CI (here: SeaweedFS in Docker)
    return os.environ.get("S3_TEST_ENDPOINT", "http://127.0.0.1:58333")


async def test_wrong_content_type_rejected(s3, s3_endpoint):
    url = await upload_link(s3, "u1")
    async with httpx.AsyncClient() as http:
        resp = await http.put(url, content=b"<html>", headers={"Content-Type": "text/html"})
    assert resp.status_code == 403

Run the same tests occasionally against a real bucket in a sandbox account, since compatible servers differ from S3 in details. The broader point applies to any SDK work covered in Cloud SDKs & Object Storage: a mock is a model of the service, and a model only checks what it implements.

Verify: a test that expects a 403 for a mismatched content type fails against a server that does not validate, and passes against one that does.

Which presigned request should this be? A decision on What will the client do with 4 outcomes. Which presigned request should this be? What will the client do? download one object presigned GET, ~300 s 403 after expiry upload, known type presigned PUT + ContentType wrong type: 403 upload, size limit needed presigned POST + content-length-range 50 KB: 400 need thousands of URLs sign in a thread or yield lag 10-59 ms Every limit must be in the signature or the policy; the client cannot be trusted with it.

Verification

Presigned URLs are generated correctly when:

  • One shared client signs with SigV4, after your handler has checked authorisation.
  • Large batches are signed off the loop or with periodic yields.
  • Uploads carry their limits in the signature or policy: content type in a PUT, size in a POST.
  • Expiries are short, under seven days, and tested against a server that enforces them.

Diagnostic Hook: count presigned URLs issued per endpoint and compare with the storage access logs' requests for those keys. Many URLs issued and few used means signing work, and leak exposure, for nothing; storage errors of 403 SignatureDoesNotMatch spiking after a client release usually means the client changed a header that was signed.

Pitfalls & edge cases

  • Signing in a tight loop. Measured: the loop stalled for the entire 1.9–2.2 s.
  • A client per URL. Measured: 2.6–3.6 ms each instead of about 0.2 ms.
  • PUT URLs as a size limit. They cannot express one; use POST with content-length-range.
  • Testing against moto. It accepted wrong types, expired URLs and oversized uploads.

Frequently Asked Questions

Does generating a presigned URL call S3?

No. It is local signing: a URL was produced against an endpoint with nothing listening. Errors such as missing keys or denied access appear only when the client uses the URL.

Why does generating many presigned URLs block my asyncio app?

aiobotocore's generate_presigned_url never suspends, so a loop of awaits runs without yielding: 10,000 URLs stalled the loop for about 2 s. Yield every 100 or sign in a thread.

How do I limit upload size with a presigned URL?

Use generate_presigned_post with a content-length-range condition; a 50 KB upload against a 10 KB limit was rejected with 400 EntityTooLarge. Presigned PUT URLs cannot limit size.

What is the maximum expiry for a presigned URL?

Seven days with SigV4; 604,801 seconds was rejected. URLs signed with temporary credentials stop working when those credentials expire.