Skip to content

Exporting Prometheus Metrics from asyncio

Standard HTTP metrics — request rate, error rate, latency — tell you that an async service is slow. They do not tell you why, because the two most common causes are invisible to them: something is blocking the event loop, or tasks are accumulating faster than they complete. Both are one gauge each, both cost almost nothing to sample, and neither exists in a synchronous service. Recording is cheap enough to be a non-issue — a labelled counter increment measured at 0.84 µs and a histogram observation at 0.93 µs — while the real constraint is cardinality: adding 5,000 distinct label values turned a 1.5 KiB scrape into 774 KiB.

Prerequisites

  • Python 3.11+ with prometheus_client (pip install prometheus-client); measurements use 0.26.
  • Loop lag measurement from health and readiness probes.
  • An ASGI app to expose the endpoint, or the library's own HTTP server.
The metrics an async service needs that a sync one does not A grid of 5 rows by 2 columns. The metrics an async service needs that a sync one does not metric type reveals event_loop_lag_seconds gauge blocking work on the loop asyncio_tasks gauge task leaks and unbounded spawning queue_depth gauge, per queue workers falling behind executor_queued gauge blocking calls crowding the thread pool inflight_requests gauge whether concurrency is real or serialised None of these exist in a threaded service; all of them are the first thing to check in an async one.

1. Define metrics once, at module scope

Metric objects register themselves, so creating one per request raises Duplicated timeseries — and creating one per label value is the cardinality mistake in disguise:

REQUESTS = Counter("http_requests_total", "Requests", ["method", "path", "status"])
LATENCY = Histogram("http_request_seconds", "Latency", ["path"],
                    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0))
LOOP_LAG = Gauge("event_loop_lag_seconds", "Event loop lag")
TASKS = Gauge("asyncio_tasks", "Live asyncio tasks")
QUEUE_DEPTH = Gauge("queue_depth", "Queue depth", ["name"])

Bucket boundaries deserve a thought, because they cannot be changed retroactively: put several around your latency target so the percentiles near it are accurate, and one comfortably above your timeout so you can see the requests that hit it.

Verify: /metrics lists each metric once, with the label sets you expect.

2. Sample the loop's own health

The two gauges that make an async service diagnosable come from one small task:

async def sample_loop_health(interval: float = 0.05) -> None:
    while True:
        started = time.perf_counter()
        await asyncio.sleep(interval)
        LOOP_LAG.set(max(0.0, time.perf_counter() - started - interval))
        TASKS.set(len(asyncio.all_tasks()))

Lag is the overshoot of a sleep that should have been exact, so it measures how long the loop was busy elsewhere. During the 500-request burst in testing it read 0.128 s — the loop genuinely was saturated — and returned to near zero afterwards.

len(asyncio.all_tasks()) is the leak detector: a number that returns to a baseline is healthy, one that climbs monotonically means tasks are being created and never finishing, as covered in tracking task growth in long-running services.

Start this task in the lifespan so it is cancelled with everything else.

Verify: a deliberate time.sleep() in a handler makes the lag gauge spike and recover.

3. Record in one place

Middleware records every request without touching handlers:

class MetricsMiddleware:
    def __init__(self, app):
        self.app = app

    async def __call__(self, scope, receive, send):
        if scope["type"] != "http":
            return await self.app(scope, receive, send)
        route = scope.get("route_pattern", "unmatched")    # the TEMPLATE, never the raw path
        started = time.perf_counter()
        status = "500"

        async def wrapped_send(message):
            nonlocal status
            if message["type"] == "http.response.start":
                status = str(message["status"])
            await send(message)

        try:
            await self.app(scope, receive, wrapped_send)
        finally:
            LATENCY.labels(path=route).observe(time.perf_counter() - started)
            REQUESTS.labels(method=scope["method"], path=route, status=status).inc()

The route template — /users/{id}, not /users/12345 — is the whole cardinality story in one line, and the next section shows what happens without it.

The recording cost is negligible: 0.84 µs per counter increment and 0.93 µs per histogram observation, measured over 100,000 operations each. Against any request this is noise.

Verify: the label set on http_requests_total has tens of values, not thousands.

What instrumentation costs per call 2 bars comparing counter.labels().inc() with the others. What instrumentation costs per call counter.labels().inc() 0.84 µs histogram.labels().observe() 0.93 µs Measured over 100,000 operations each with prometheus_client 0.26. At under a microsecond, the cost is never the reason not to record a metric.

4. Treat cardinality as the hard limit

Every distinct label combination is a separate time series that lives in your process and in the backend forever. Recording a user id as a label, with 5,000 users:

bounded labels:               1,545 bytes, scraped in 2.0 ms
5,000 distinct path values:   774 KiB, 10,032 lines

A 500x increase from one label, in one process, for one metric. In the backend that is 5,000 series per instance per metric, and the usual outcome is either a very large bill or a metrics system that stops accepting writes.

The rule is simple: labels must be things you would group by, and their value set must be bounded and known in advance — method, route template, status class, queue name, outcome. Identifiers belong in logs and trace attributes, which are designed for high cardinality, as described in tracing asyncio services with OpenTelemetry.

Auditing is easy: scrape /metrics, count the lines, and look at what grew.

Verify: the scrape size is stable over time rather than growing with traffic.

What one high-cardinality label does to a scrape 2 bars comparing bounded labels with the others. What one high-cardinality label does to a scrape bounded labels 1.5 KiB, 2 ms 5,000 distinct label values 774 KiB, 10,032 lines Each new label combination is a permanent time series in the process and in the backend. Cardinality is the only real limit on metrics: keep user and request ids out of labels.

5. Expose the endpoint, and handle multiple workers

Exposing is a route:

async def metrics_endpoint(request):
    return Response(generate_latest(REGISTRY), media_type=CONTENT_TYPE_LATEST)

The scrape took 2.0 ms for 1,545 bytes, and generation is synchronous, so a very large registry can measurably block the loop — another reason to keep cardinality bounded.

With several uvicorn workers behind one port, a scrape hits a random worker and returns that process's numbers, which makes counters look like they jump around. Two standard fixes:

  • prometheus_client.multiprocess, with PROMETHEUS_MULTIPROC_DIR set, aggregates across processes via files. It has real restrictions — gauges need a multiprocess_mode, and the directory must be cleaned between restarts.
  • One metrics port per process, scraped separately, which keeps per-worker visibility and is often simpler in container platforms where one worker per container is the norm anyway.

Either way, remember that gauges like asyncio_tasks and event_loop_lag_seconds are inherently per process — aggregating them by averaging hides the one worker that is in trouble. Use max for lag.

Verify: the scrape's values are consistent between consecutive scrapes, or you are hitting different workers.

Counters, gauges and histograms 2 columns contrasting counters and gauges, histograms. Counters, gauges and histograms counters and gauges how much, how many counters only go up rate() turns them into per-second gauges are a current value cheap, tiny, always on histograms how long, and how spread bucket boundaries chosen up front p99 computed by the backend one series per bucket per label set the main source of cardinality Pick bucket boundaries around your latency target: they cannot be changed retroactively.
Wiring metrics into an async service 5 ordered steps. Wiring metrics into an async service define metrics once module scope, one registry sample loop lag and tasks a small background task record in middleware one place, every request expose /metrics cheap, unauthenticated internally aggregate per worker each process has its own numbers With several workers behind one port, scrape each process or use the multiprocess collector.

Verification

Metrics are well set up when:

  • Loop lag and task count are exported, and alertable.
  • Metrics are defined once, at module scope.
  • Labels are bounded, with route templates rather than paths.
  • Scrape size is stable, not growing with traffic.
  • Histogram buckets bracket your latency target.
  • Multi-worker aggregation is deliberate, not accidental.

Pitfalls & edge cases

  • User or request ids as labels. 5,000 values turned a 1.5 KiB scrape into 774 KiB.
  • Creating metric objects per request. Raises Duplicated timeseries, or leaks series.
  • Raw paths as a label. Every URL variant becomes a series; use the route template.
  • Blocking work in the metrics endpoint. Generation is synchronous; keep the registry small.
  • Averaging per-worker gauges. Hides the one worker with 10 seconds of lag; use max.
  • Counters that can go down. Reset them only by restarting the process; rate() assumes monotonicity.

Frequently Asked Questions

Which metrics should an asyncio service export?

The usual rate, error and duration metrics, plus three that are specific to async: event loop lag, live task count, and the depth of any internal queue. Loop lag distinguishes "our dependency is slow" from "something is blocking the loop", and task count is how a task leak becomes visible.

How do I measure event loop lag for Prometheus?

Run a background task that records the time, sleeps a known interval, and sets a gauge to the overshoot. Measured during a 500-request burst, the gauge read 0.128 s and returned to near zero afterwards. Export it as a gauge and alert on the maximum across workers.

How much overhead do Prometheus metrics add?

Very little: 0.84 µs per labelled counter increment and 0.93 µs per histogram observation, measured over 100,000 operations each. The cost is never the reason to skip a metric; cardinality is.

Why should I not use user IDs as metric labels?

Because each distinct label combination is a permanent time series. Recording 5,000 distinct values took one scrape from 1,545 bytes to 774 KiB and 10,032 lines, in one process. Identifiers belong on trace attributes and in logs, which are built for high cardinality.

How do Prometheus metrics work with multiple uvicorn workers?

Each process has its own registry, so a scrape returns one worker's numbers. Either use prometheus_client's multiprocess mode with PROMETHEUS_MULTIPROC_DIR, or expose one metrics port per process and scrape them separately — which also preserves the per-worker gauges you actually want.