Skip to content

Sending Async HTTP Requests Through Proxies

Corporate networks, scraping setups and egress-controlled clusters send outbound HTTP through a proxy, and the two main async clients disagree about when to use one. Measured on Python 3.14 with httpx 0.28.1, aiohttp 3.14.3 and a local proxy.py 2.4.10 forward proxy requiring basic authentication: with HTTP_PROXY set in the environment, httpx sent requests through the proxy by default — the target saw Via: 1.1 proxy.py v2.4.10 — while aiohttp went direct unless the session had trust_env=True. A missing proxy password produced a 407 response object for plain HTTP in both clients, but an exception for HTTPS: httpx.ProxyError and aiohttp.ClientHttpProxyError. Through the proxy, sequential requests with aiohttp took 0.27 ms instead of 0.09 ms for HTTP and 0.24 ms instead of 0.13 ms for HTTPS, and 50 concurrent requests ran at 16,697 per second instead of 22,765 for HTTP. This guide configures both clients explicitly and handles the failure modes.

Prerequisites

1. Find out which client honours proxy variables

Before configuring anything, test what each client does with the environment the service will run in. Point a request at a target that echoes the Via header a proxy adds:

async with httpx.AsyncClient() as c:
    print("httpx", (await c.get(URL)).json()["via"])

async with aiohttp.ClientSession() as s:
    async with s.get(URL) as r:
        print("aiohttp", (await r.json())["via"])

async with aiohttp.ClientSession(trust_env=True) as s:
    async with s.get(URL) as r:
        print("aiohttp trust_env", (await r.json())["via"])

Measured with HTTP_PROXY=http://user:secret@127.0.0.1:58510: httpx went through the proxy, since its trust_env defaults to True; httpx with trust_env=False went direct; aiohttp with default settings went direct; aiohttp with trust_env=True went through the proxy. Adding NO_PROXY=127.0.0.1 made all four go direct. A service that works in a developer's shell can therefore bypass the proxy in production — or the reverse — depending only on which library it uses.

Verify: a request to an echo endpoint shows the proxy's Via header, or its absence, as intended in each environment.

HTTP_PROXY set in the environment A grid of 4 rows by 3 columns. HTTP_PROXY set in the environment client HTTP_PROXY only HTTP_PROXY + NO_PROXY=127.0.0.1 httpx, defaults (trust_env=True) via proxy direct httpx, trust_env=False direct direct aiohttp, defaults (trust_env=False) direct direct aiohttp, trust_env=True via proxy direct httpx 0.28.1, aiohttp 3.14.3; detected by the proxy's Via header.

2. Configure the proxy explicitly

Relying on environment variables makes behaviour depend on deployment details. Pass the proxy in code, read from the service's own configuration:

# httpx: one proxy for all traffic, with exceptions by URL pattern
client = httpx.AsyncClient(
    proxy=settings.proxy_url,                      # "http://user:pass@proxy:3128"
    mounts={"http://internal.svc": None},          # internal hosts go direct
    trust_env=False,                               # ignore stray env vars
    timeout=httpx.Timeout(10.0, connect=3.0),
)

# aiohttp: a session-wide default, overridable per request
session = aiohttp.ClientSession(proxy=settings.proxy_url, trust_env=False)

The httpx mounts dictionary maps URL patterns to transports; None means no proxy for that pattern, and a separate httpx.AsyncHTTPTransport(proxy=...) can send some hosts through a different proxy. aiohttp accepts proxy= on the session and on each request, and proxy_auth=aiohttp.BasicAuth(...) as an alternative to credentials in the URL. Keep credentials out of logs: a proxy URL with embedded credentials is easy to log by accident, so log only the proxy host, or pass credentials separately with proxy_auth.

Verify: with all proxy environment variables unset, requests still go through the configured proxy; with them set to a wrong value, nothing changes.

3. Handle proxy authentication failures

A forward proxy answers bad credentials with 407 Proxy Authentication Required, and how that reaches your code depends on the scheme:

try:
    r = await client.get(url)
    if r.status_code == 407:                       # plain HTTP: a normal response
        raise ProxyAuthFailed(r.headers.get("Proxy-Authenticate"))
except httpx.ProxyError as e:                      # HTTPS: CONNECT refused
    raise ProxyAuthFailed(str(e)) from e

Measured with no credentials: for http:// URLs, both httpx and aiohttp returned a response with status 407 — the proxy forwarded nothing and answered for the target. For https:// URLs, the client must first open a tunnel with CONNECT, and both raised: httpx ProxyError: 407 Proxy Authentication Required, aiohttp ClientHttpProxyError: 407. Code that only checks for exceptions misses the HTTP case and treats a 407 as the target's answer; code that only checks status codes misses the HTTPS case. Neither should be retried: a 407 does not fix itself, and retrying it only adds load on the proxy.

Verify: a test with wrong proxy credentials raises the same application error for HTTP and HTTPS targets, and the retry policy excludes it.

An HTTPS request through a forward proxy A sequence of 6 messages between 3 participants. An HTTPS request through a forward proxy client proxy target CONNECT target:443 + Proxy-Authorization TCP connect 200 Connection established TLS handshake + GET, through the tunnel response, tunnel kept for reuse no credentials: 407 -> exception For plain HTTP the proxy forwards each request and a 407 is a response.

4. Measure the proxy's cost

A proxy adds a network hop and the proxy's own processing. Measure both sequential latency and concurrent throughput, with connections reused:

async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(limit=50)) as s:
    t = time.perf_counter()
    await asyncio.gather(*(fetch(s, url, proxy=PROXY) for _ in range(2000)))
    rate = 2000 / (time.perf_counter() - t)

Measured locally with aiohttp: sequential HTTP requests took 0.09 ms direct and 0.27 ms through the proxy; HTTPS took 0.13 ms and 0.24 ms, as the CONNECT tunnel was opened once and reused. With 50 concurrent requests, HTTP ran at 22,765 per second direct and 16,697 through the proxy, HTTPS at 14,992 and 11,520. httpx showed the same pattern in sequential requests, 0.30 ms against 0.45 ms for HTTP, but with 50 concurrent requests its own client overhead held it to 440–505 per second either way, as measured in migrating requests code to httpx incrementally. On a real network, the hop to the proxy and the proxy's capacity dominate these local figures, so measure in the target environment.

Verify: a load test through the production proxy shows the added latency per request and the proxy's maximum throughput, and both fit the service's budget.

aiohttp, 50 concurrent requests per second 4 horizontal bars comparing HTTP, direct with the others. aiohttp, 50 concurrent requests per second HTTP, direct 22,765/s HTTP, via proxy 16,697/s HTTPS, direct 14,992/s HTTPS, via proxy (CONNECT) 11,520/s Local proxy.py with 2 workers and basic auth; tunnels reused. Sequential latency rose from 0.09 to 0.27 ms for HTTP.

5. Keep tunnels and credentials healthy

Two operational details matter once a proxy is in the path. First, connection reuse: each new HTTPS connection through a proxy costs a CONNECT exchange plus a TLS handshake, so a shared client is even more important than without a proxy. Second, failures at the proxy look like network failures to the client:

RETRYABLE = (httpx.ConnectError, httpx.ReadTimeout)          # proxy or target unreachable
NOT_RETRYABLE = (httpx.ProxyError,)                          # 407, or proxy refused CONNECT

async def get_with_retry(client, url, attempts=3):
    for attempt in range(attempts):
        try:
            return await client.get(url)
        except NOT_RETRYABLE:
            raise
        except RETRYABLE:
            if attempt == attempts - 1:
                raise
            await asyncio.sleep(0.2 * 2 ** attempt)

Separate proxy refusals from transient connection errors in retry logic, and log which hop failed. When the proxy rotates credentials, rebuild the client rather than mutating it, so that pooled tunnels opened with the old credentials are closed. For a full retry policy, see retrying httpx requests with transport retries.

Verify: a rotation of proxy credentials is followed by a client rebuild, and the retry policy never retries ProxyError.

Verification

Proxy use is correct when:

  • The proxy is configured in code, with trust_env=False, so environment variables cannot silently change routing.
  • Bypass rules are explicit — mounts in httpx, per-request proxy=None in aiohttp — and tested.
  • 407 is handled for both schemes: a status for HTTP, an exception for HTTPS.
  • The proxy's latency and throughput are measured in the target environment.

Diagnostic Hook: when an aiohttp service cannot reach the internet in an environment where curl works, compare the environment: curl and httpx read HTTPS_PROXY, aiohttp does not unless the session sets trust_env=True. In this test, aiohttp with defaults went direct while HTTP_PROXY was set.

Pitfalls & edge cases

  • Relying on aiohttp to read proxy variables. It went direct by default.
  • Relying on httpx to ignore them. It used the proxy by default.
  • Checking only exceptions for proxy auth. Plain HTTP returned a 407 response.
  • Logging the proxy URL. It contains the credentials.

Frequently Asked Questions

Does aiohttp use the HTTP_PROXY environment variable?

Only with trust_env=True on the ClientSession. With defaults, a request went direct while HTTP_PROXY was set; httpx used the proxy by default.

How do I set a proxy in httpx?

Pass proxy="http://user:pass@host:port" to AsyncClient, use mounts to send some URL patterns direct or elsewhere, and set trust_env=False to ignore environment variables.

Why do I get a 407 status instead of an exception?

For plain HTTP the proxy answers the request itself, so 407 arrives as a response. For HTTPS, the CONNECT fails and httpx raises ProxyError, aiohttp ClientHttpProxyError.

How much latency does an HTTP proxy add?

Locally, aiohttp went from 0.09 to 0.27 ms per HTTP request and from 0.13 to 0.24 ms for HTTPS with a reused tunnel. Measure the real proxy hop in production.