Skip to content

Migrating requests Code to httpx Incrementally

A codebase built on requests cannot become asynchronous in one change, and does not need to. The migration can proceed in steps that each ship on their own: run existing calls in threads, switch the HTTP library to httpx while staying synchronous, then convert hot paths to httpx.AsyncClient. Each step has a behaviour change or a performance trap worth measuring. Measured on Python 3.14 with requests 2.34.2 and httpx 0.28.1 against a local API answering in 50 ms: sequential requests calls ran at 19 req/s. Running the same function through asyncio.to_thread with 400 concurrent calls reached 511 req/s, capped by the default executor's 28 threads. httpx did not follow a redirect that requests followed — 302 against 200 — and timed out a hanging request after 5.0 s, where requests without a timeout waited the full 10 s for the server. With a shared AsyncClient, throughput depended on the connection pool size in an unexpected way: 189 req/s with 10 connections, 376 with 20, 600 with 50, and only 114 with 100, where the client spent 8.67 ms of CPU per request and ran at 99% CPU. This guide walks through the steps and their checks.

Prerequisites

1. Unblock the loop first with to_thread

When async code needs an existing requests-based function, the first step changes no library code: call it in a worker thread.

session = requests.Session()

def fetch_item(item_id: int) -> dict:                 # existing code, unchanged
    r = session.get(f"{BASE}/items/{item_id}", timeout=10)
    r.raise_for_status()
    return r.json()

async def handler(item_ids):
    return await asyncio.gather(*(asyncio.to_thread(fetch_item, i) for i in item_ids))

Measured with 400 concurrent calls: 511 requests per second, against 19 when the same calls ran one after another — and the event loop stayed free throughout. The ceiling is the default executor's size, min(32, cpu_count + 4), which was 28 here: 28 threads × one 50 ms call each is about 540 per second. That is often enough to ship, and it buys time for the rest of the migration. requests.Session is not documented as thread-safe for concurrent use; sharing one across threads worked in this test, but a session per thread or a lock around configuration changes avoids surprises.

Verify: loop lag stays low while requests calls run, and the executor size is known to be the throughput ceiling.

400 calls to a 50 ms API A grid of 6 rows by 3 columns. 400 calls to a 50 ms API approach req/s limit requests, sequential 19 one call at a time requests via asyncio.to_thread 511 28 executor threads httpx.AsyncClient, pool 20 376 20 connections x 50 ms httpx.AsyncClient, pool 50 600 client CPU 73% httpx.AsyncClient, pool 100 114 client CPU 99%, 8.67 ms/request new AsyncClient per request 266 new connection each time requests 2.34.2, httpx 0.28.1, httpcore 1.0.9, Python 3.14.

2. Switch to httpx's sync client and check behaviour differences

httpx's synchronous Client is close to a requests.Session, so the library can change before the concurrency model does. The differences that bite are defaults, not method names:

with httpx.Client(base_url=BASE, follow_redirects=True, timeout=10.0) as client:
    r = client.get("/old-items")
    r.raise_for_status()

Measured against the same server: a request to an endpoint returning 302 Found came back as 200 from /items/1 with requests — which follows redirects by default — and as 302 from httpx, whose follow_redirects defaults to False. A request to an endpoint that never answered raised httpx.ReadTimeout after 5.0 s, httpx's default timeout, while requests, which has no default timeout, waited the full 10 s until the server finally responded. Other differences to check: requests' params and data encoding, the exception hierarchy (httpx.HTTPStatusError against requests.HTTPError), and environment proxy handling. Write a small suite of calls against the real APIs you use, run it with both clients, and compare status codes and bodies.

Verify: the same request suite returns identical status codes and bodies under requests and httpx, with explicit redirect and timeout settings.

3. Convert hot paths to a shared AsyncClient

Once code uses httpx, converting a function to async is mostly adding async and await and swapping Client for AsyncClient. Create one client for the application's lifetime and share it:

@asynccontextmanager
async def lifespan(app):
    async with httpx.AsyncClient(
        base_url=BASE,
        follow_redirects=True,
        timeout=httpx.Timeout(10.0, connect=3.0),
        limits=httpx.Limits(max_connections=20, max_keepalive_connections=20),
    ) as client:
        app.state.http = client
        yield

async def fetch_item(client: httpx.AsyncClient, item_id: int) -> dict:
    r = await client.get(f"/items/{item_id}")
    r.raise_for_status()
    return r.json()

Measured: a new AsyncClient per request ran at 266 req/s — every request opened a fresh connection — so the shared client is not optional. The limits deserve measuring, described next.

Verify: one AsyncClient instance is created at start-up and closed at shutdown; no request path constructs one.

4. Size the connection pool by measurement

The surprising result of this migration test was that a larger httpx connection pool was slower. With a warmed pool and at most as many requests in flight as connections:

for conns in (10, 20, 50, 100):
    limits = httpx.Limits(max_connections=conns, max_keepalive_connections=conns)
    # ... 400 requests, `conns` at a time, measuring wall time and process CPU time

Measured: 10 connections gave 189 req/s using 0.60 ms of client CPU per request; 20 gave 376 and 0.52 ms; 50 gave 600 and 1.21 ms, with the client at 73% CPU; 100 gave 114 req/s, with 8.67 ms of CPU per request and the client at 99% CPU — the client process itself had become the bottleneck, and adding connections made each request more expensive. For comparison, aiohttp with a 100-connection limit against the same server ran at 1,582 req/s. This is a measured property of httpx 0.28.1 with httpcore 1.0.9 on this machine, not a law; re-measure on your versions, and size the pool from a sweep like this rather than by guessing that bigger is better. For services that need hundreds of concurrent requests per process, compare clients as in asyncio vs threading for 1000 concurrent HTTP requests.

Verify: a pool-size sweep shows where throughput peaks and client CPU stays well below 100%.

httpx.AsyncClient throughput by pool size, warmed pool 4 horizontal bars comparing pool 10 (0.60 ms CPU/req) with the others. httpx.AsyncClient throughput by pool size, warmed pool pool 10 (0.60 ms CPU/req) 189/s pool 20 (0.52 ms CPU/req) 376/s pool 50 (1.21 ms CPU/req) 600/s pool 100 (8.67 ms CPU/req) 114/s aiohttp with limit=100 against the same server: 1,582 req/s. More connections cost more client CPU per request here.

5. Keep sync callers working during the transition

During the migration, some callers are async and some are not. Keep one implementation of the request logic and give it both faces, rather than maintaining two:

def build_request(item_id: int) -> tuple[str, dict]:
    return f"/items/{item_id}", {"params": {"expand": "owner"}}

def parse(r: httpx.Response) -> dict:
    r.raise_for_status()
    return r.json()

async def fetch_item_async(client: httpx.AsyncClient, item_id: int) -> dict:
    path, kw = build_request(item_id)
    return parse(await client.get(path, **kw))

def fetch_item_sync(client: httpx.Client, item_id: int) -> dict:
    path, kw = build_request(item_id)
    return parse(client.get(path, **kw))

Request construction and response parsing are pure functions shared by both; only the transport call differs. When the last synchronous caller is gone, delete the sync face. Larger libraries can generate the sync version from the async one, as described in offering sync and async versions of one API.

Verify: sync and async paths share request building and parsing, and both are covered by the same tests.

An incremental migration from requests A flow of 5 stages. An incremental migration from requests to_thread unblock the loop, no code change httpx.Client explicit redirects + timeouts AsyncClient shared, created at start-up Pool sweep find the throughput peak Shared core sync face until callers move Each step ships and is measured on its own.

Verification

The migration is on track when:

  • No requests call runs on the event loop; they go through to_thread until converted.
  • httpx clients set redirects and timeouts explicitly, after comparing responses with requests.
  • One AsyncClient is shared, with a pool size chosen from a measured sweep.
  • Request building and parsing are shared between sync and async callers until the sync ones are gone.

Diagnostic Hook: when an endpoint's latency rises after moving to AsyncClient, check the process's CPU before the network. A client spending milliseconds of CPU per request — 8.67 ms in the 100-connection test — shows up as latency that grows with concurrency while the remote service's own timings stay flat.

Pitfalls & edge cases

  • Assuming httpx follows redirects. Measured: 302 where requests returned 200.
  • Relying on requests' lack of a timeout. httpx raised after 5 s by default.
  • A new AsyncClient per request. Measured: 266 req/s.
  • Bigger connection pools by default. Measured: 100 connections ran at 114 req/s against 600 for 50.

Frequently Asked Questions

How do I migrate from requests to httpx?

In steps: call existing requests code through asyncio.to_thread, switch to httpx.Client with explicit follow_redirects and timeout after comparing responses, then convert hot paths to a shared httpx.AsyncClient.

Does httpx follow redirects like requests?

No: follow_redirects defaults to False. A 302 came back as 302 from httpx and as 200 after redirection from requests.

What is httpx's default timeout?

Five seconds; a hanging request raised ReadTimeout after 5.0 s, while requests, which has no default timeout, waited 10 s for the server.

How many connections should an httpx AsyncClient have?

Measure: in testing, throughput peaked at 50 connections (600 req/s) and collapsed at 100 (114 req/s) as client CPU per request rose to 8.67 ms.