Skip to content

Bounding In-Memory Async Caches by Size

Every in-process cache in an async service needs a bound, and almost every one gets maxsize=10_000. That bounds the number of entries, not the memory they use, and the gap is enormous. Measured with tracemalloc, a cached dict of 10 short fields cost 1.3 KiB per entry; one with 50 fields of 100 characters cost 10.9 KiB; one with 200 fields of 1,000 characters cost 219.5 KiB. The same maxsize=10_000 is therefore anywhere from 13 MB to 2.1 GB. sys.getsizeof() does not help: for the largest of those dicts it reported 6,576 bytes, because it measures the dict's own table and not the strings it points to. This guide bounds caches by estimated bytes, chooses an estimator that tracks reality, and verifies it.

Prerequisites

1. Measure what an entry really costs

Before choosing a bound, measure your actual values. tracemalloc counts every allocation, so a snapshot diff around building the cache gives the real per-entry cost:

import json
import sys
import tracemalloc


def per_entry_cost(build_value, n: int = 1000) -> float:
    tracemalloc.start()
    before = tracemalloc.take_snapshot()
    cache = {f"k{i}": build_value(i) for i in range(n)}
    after = tracemalloc.take_snapshot()
    tracemalloc.stop()
    total = sum(s.size_diff for s in after.compare_to(before, "filename"))
    sample = next(iter(cache.values()))
    print(f"traced {total / n / 1024:.1f} KiB/entry, "
          f"getsizeof {sys.getsizeof(sample)} B, json {len(json.dumps(sample))} B")
    return total / n

Results for three value shapes:

Value shape Traced per entry sys.getsizeof JSON length
10 fields × 10 chars 1.3 KiB 272 B 250 B
50 fields × 100 chars 10.9 KiB 1,584 B 5,790 B
200 fields × 1,000 chars 219.5 KiB 6,576 B 203,290 B

Run this against real production values, sampled — not invented ones. The traced figure is the number your bound should control.

Verify: the traced cost for your values is recorded somewhere the cache configuration can refer to.

Memory per cached entry, by value shape 3 horizontal bars comparing 200 fields x 1,000 chars with the others. Memory per cached entry, by value shape 200 fields x 1,000 chars 219.5 KiB 50 fields x 100 chars 10.9 KiB 10 fields x 10 chars 1.3 KiB Measured with tracemalloc snapshot diffs over 1,000 entries; Python 3.14. maxsize=10,000 means 13 MB for the first shape and 2.1 GB for the last.

2. Pick an estimator that tracks the real cost

Measuring every entry with tracemalloc is far too slow for the hot path. Pick a cheap proxy that grows with the real cost. From the table, sys.getsizeof is useless for nested data; the length of a serialised form tracks the large cases well and underestimates small ones, which is the safe direction to be wrong in only if you add a per-entry overhead:

def estimate_bytes(value) -> int:
    if isinstance(value, (bytes, bytearray, memoryview)):
        return len(value) + 64
    if isinstance(value, str):
        return len(value) + 64
    try:
        return len(json.dumps(value, separators=(",", ":"))) + 1024   # + per-entry overhead
    except (TypeError, ValueError):
        return 4096                                                   # unknown: assume a page

If values are already serialised — cached HTTP bodies, Redis payloads, pickled objects — their length is an excellent estimate and free to compute. For values computed from a database row, the estimator can often be a constant per type, measured once with step 1. Serialising purely to estimate size costs time on every insert; avoid it for large, frequently replaced values.

Verify: for a sample of real values, estimate_bytes() is within a factor of two of the traced cost, never wildly under.

3. Evict by bytes, not by count

An LRU that tracks total estimated bytes evicts until it is under its budget:

import time
from collections import OrderedDict


class ByteBoundedLRU:
    def __init__(self, max_bytes: int, ttl: float | None = None, estimate=estimate_bytes) -> None:
        self.max_bytes, self.ttl, self.estimate = max_bytes, ttl, estimate
        self._d: OrderedDict[str, tuple[float, int, object]] = OrderedDict()
        self.bytes = 0
        self.evictions = 0

    def get(self, key: str):
        item = self._d.get(key)
        if item is None:
            return None
        expires, _, value = item
        if self.ttl is not None and expires < time.monotonic():
            self._drop(key)
            return None
        self._d.move_to_end(key)
        return value

    def put(self, key: str, value) -> None:
        size = self.estimate(value)
        if size > self.max_bytes // 10:          # never let one entry take over the cache
            return
        if key in self._d:
            self._drop(key)
        expires = time.monotonic() + self.ttl if self.ttl else float("inf")
        self._d[key] = (expires, size, value)
        self.bytes += size
        while self.bytes > self.max_bytes:
            oldest = next(iter(self._d))
            self._drop(oldest)
            self.evictions += 1

    def _drop(self, key: str) -> None:
        _, size, _ = self._d.pop(key)
        self.bytes -= size

The max_bytes // 10 guard refuses values larger than a tenth of the budget. Without it, one huge value evicts thousands of small hot ones and is often itself a one-off read. All operations are synchronous, so on one event loop they cannot interleave; no lock is needed unless the cache is touched from threads.

Verify: after inserting values of mixed sizes, cache.bytes never exceeds max_bytes, and RSS growth tracks it.

What happens on put() in a byte-bounded LRU A flow of 4 stages. What happens on put() in a byte-bounded LRU estimate bytes cheap proxy refuse if > budget/10 protect hot entries insert, add to total move to end evict oldest until under budget The budget is in bytes, so one large value costs what it actually costs.

4. Set the budget from the container, not a guess

The cache budget is a slice of the process's memory limit, alongside everything else the worker needs. Derive it from the limit and the number of caches:

import os


def container_memory_limit() -> int | None:
    for path in ("/sys/fs/cgroup/memory.max", "/sys/fs/cgroup/memory/memory.limit_in_bytes"):
        try:
            raw = open(path).read().strip()
            if raw != "max":
                return int(raw)
        except (OSError, ValueError):
            continue
    return None


limit = container_memory_limit() or 2 * 1024**3
user_cache = ByteBoundedLRU(max_bytes=int(limit * 0.15), ttl=30)      # 15% of the container

Fifteen percent across all caches is a reasonable starting point for a typical API worker; heap fragmentation and the gap between the estimate and the real cost need headroom. With several workers per container — uvicorn's --workers, a process pool — divide by the worker count, since each holds its own copy, as discussed in sizing uvicorn workers for async services.

Verify: run a load test that fills the cache; peak RSS stays well under the container limit, and the cache reports bytes near max_bytes.

5. Watch hit rate against size

A bound is a trade between memory and hit rate. Measure both, and change the bound based on the curve rather than intuition:

class InstrumentedLRU(ByteBoundedLRU):
    def __init__(self, *a, **kw) -> None:
        super().__init__(*a, **kw)
        self.hits = self.misses = 0

    def get(self, key):
        v = super().get(key)
        if v is None:
            self.misses += 1
        else:
            self.hits += 1
        return v


def report(c: InstrumentedLRU) -> dict:
    total = c.hits + c.misses or 1
    return {"hit_ratio": c.hits / total, "bytes": c.bytes,
            "entries": len(c._d), "evictions": c.evictions}

If doubling the budget barely moves the hit ratio, the working set already fits and the extra memory is wasted. If evictions are high and the hit ratio is low, the working set is larger than the cache, and a shared tier such as Redis — rather than a bigger local cache multiplied by every worker — is usually the better spend, as in building a two-tier local and Redis cache.

Verify: you have a hit-ratio measurement at the current budget and at least one other budget.

How should this cache be bounded? A decision on How do value sizes behave with 3 outcomes. How should this cache be bounded? How do value sizes behave? small and uniform maxsize count is fine measure once varied or large byte budget + estimator refuse huge values working set too big shared Redis tier not N local copies Count limits are only safe when every entry costs about the same.

Verification

The cache is properly bounded when:

  • Per-entry cost is measured for real values, with tracemalloc.
  • The estimator tracks real cost within a small factor and never vastly under.
  • The budget derives from the container limit and the number of workers.
  • Hit ratio and evictions are exported, and the budget is tuned from them.

Diagnostic Hook: export bytes, entries, evictions and hit_ratio per cache, alongside process RSS. Alert if RSS grows while every cache sits at its budget — that is memory outside the caches, and the caches are not the leak; alert if a cache's eviction rate climbs while its hit ratio falls, which means the working set has outgrown it.

Pitfalls & edge cases

  • sys.getsizeof as the estimator. It ignores everything a container points to.
  • No per-entry overhead in the estimate. Thousands of tiny entries cost far more than their payload.
  • Caching references to large shared objects. The cache's estimate counts them, but they may be kept alive elsewhere anyway; estimate what the cache actually adds.
  • Per-worker budgets that ignore worker count. Eight workers × 15% is 120% of the container.

Frequently Asked Questions

Does maxsize bound the memory of a Python cache?

No, only the number of entries. Measured per-entry costs ranged from 1.3 KiB to 219.5 KiB depending on the value, so 10,000 entries could be 13 MB or 2.1 GB.

Why is sys.getsizeof wrong for cache values?

It reports the size of the object itself, not of the objects it references. For a dict of long strings it reported 6,576 bytes where tracemalloc measured 219.5 KiB.

How do I limit a Python cache by memory?

Estimate each value's size with a cheap proxy such as its serialised length plus a fixed overhead, keep a running total in an LRU, and evict the least recently used entries until the total is under a byte budget derived from the container's memory limit.

How big should an in-process cache be?

Start with a slice of the container's memory, such as 15% divided by workers per container, then adjust using measured hit ratio and eviction rate at different budgets.