Skip to content

Keeping Headless Browser Memory Bounded

Long-running browser automation — a crawler that runs all day, a rendering service, a monitoring bot — fails most often by memory, not by errors: Chromium's footprint grows until the container is killed. How it grows depends on how pages are used, and the measurement that matters is proportional set size (PSS) across all of Chromium's processes, since its processes share memory and summed RSS double-counts it. Measured on Python 3.14 with Playwright 1.63 and headless Chromium, eight workers processing JavaScript-rendered pages from a local test site: with a fresh context per page, PSS sampled every 200 pages over 1,000 pages stayed between 303 and 336 MiB with no upward trend, at 13.7 pages per second. With one page per worker reused for every navigation, PSS reached 641 MiB after 100 pages and 906 MiB after 400. Closing everything brought the browser back to about 190–200 MiB, and recycling the whole browser took 145 ms. This guide keeps memory in a band for as long as the process runs.

Prerequisites

1. Measure Chromium's memory with PSS

Measure the browser's processes, not the Python process — Python's own memory is a small part of the total:

import os
import psutil


def chromium_pss_mib(parent: psutil.Process | None = None) -> float:
    parent = parent or psutil.Process(os.getpid())
    total = 0
    for child in parent.children(recursive=True):
        try:
            if "chrom" in child.name() or "headless" in child.name():
                total += child.memory_full_info().pss          # Linux: shared pages split fairly
        except (psutil.NoSuchProcess, psutil.AccessDenied):
            pass
    return total / 2**20

On the test machine the idle browser measured 174 MiB of PSS and 321 MiB of summed RSS — the difference is memory shared between Chromium's processes, counted once per process by RSS. Container memory limits charge something closer to PSS, so it is the figure to plan with. Walking the process tree takes tens of milliseconds, so sample it from a separate periodic task (or a thread), not inside the page-processing loop; doing it per page measurably slowed the crawl in testing.

Verify: your memory metric for the browser is PSS-based and sampled on a timer.

Chromium PSS over a long run, 8 workers A grid of 3 rows by 3 columns. Chromium PSS over a long run, 8 workers page lifecycle PSS during the run trend fresh context per page (1,000 pages) 303-336 MiB flat page per URL in one shared context (400) 300-324 MiB flat one page per worker, reused (400) 641 -> 906 MiB rising Playwright 1.63, Python 3.14; samples every 100-200 pages.

2. Throw pages away instead of reusing them

The measured difference comes from what a page accumulates across navigations — its JavaScript heap, history, caches and back-forward state. A page that is closed after each URL takes all of that with it:

async def worker(browser, queue: asyncio.Queue) -> None:
    while not queue.empty():
        url = queue.get_nowait()
        async with await browser.new_context() as context:       # everything freed on exit
            page = await context.new_page()
            await page.goto(url)
            await process(page)

A fresh context per page held PSS flat over 1,000 pages. Pages created and closed per URL inside one long-lived context also stayed flat (300–324 MiB), so the decisive factor is closing the page, not necessarily the context; a fresh context adds isolation of cookies and storage for about 5 ms. The reused-page version — the one that looks most economical — was the only one that grew, and it was also slower, at 7.6 pages per second against 12.0. The cleanup side of the same discipline, for jobs that are cancelled midway, is in cleaning up Playwright on cancellation.

Verify: no worker keeps a page open across more than one job.

3. Bound memory by bounding concurrency

With pages freed after each job, memory is a function of how many are open at once: about 174 MiB for the browser plus roughly 20 MiB per open context with a loaded page on the test site. Size the worker pool from the memory budget as well as from throughput:

BROWSER_BASE_MIB = 175
PER_PAGE_MIB = 20            # measure on your real pages: heavy sites use far more
HEADROOM = 0.6               # keep 40% spare for heavy pages and spikes


def max_workers(limit_mib: int) -> int:
    return max(1, int((limit_mib * HEADROOM - BROWSER_BASE_MIB) / PER_PAGE_MIB))

print(max_workers(1024))     # 21 for a 1 GiB container on the test pages

Real pages vary enormously — a media-heavy site can use hundreds of megabytes per tab — so measure the per-page figure on a sample of the actual targets, take a high percentile rather than the mean, and keep headroom. The throughput plateau measured in pooling browser contexts for concurrent pages was reached at eight workers; there is no reason to run more than the smaller of the two limits.

Verify: peak PSS under the chosen worker count stays below 60–70% of the container's memory limit.

Chromium PSS every 200 pages, context per page 5 horizontal bars comparing after 200 pages with the others. Chromium PSS every 200 pages, context per page after 200 pages 334 MiB after 400 pages 329 MiB after 600 pages 303 MiB after 800 pages 336 MiB after 1,000 (draining) 199 MiB 8 workers, 13.7 pages/s; the last sample was taken as workers finished. No drift: memory tracked open pages, not pages processed.

4. Recycle the browser on a schedule anyway

Even with flat averages, long-lived browser processes can accumulate state the page lifecycle does not reach — GPU and font caches, compiled code, the occasional leak in a site's service worker. Restarting the browser periodically is cheap insurance:

class RecyclingBrowser:
    def __init__(self, playwright, every: int = 500) -> None:
        self.p, self.every, self.used, self.browser = playwright, every, 0, None
        self._lock = asyncio.Lock()

    async def get(self):
        async with self._lock:
            if self.browser is None or self.used >= self.every:
                old = self.browser
                self.browser, self.used = await self.p.chromium.launch(), 0
                if old is not None:
                    asyncio.get_running_loop().create_task(self._close_when_idle(old))
            self.used += 1
            return self.browser

    async def _close_when_idle(self, browser) -> None:
        while browser.contexts:                       # let in-flight jobs finish
            await asyncio.sleep(1)
        await browser.close()

Closing and relaunching took 145 ms in testing — negligible against a schedule of hundreds of pages. Launching the replacement before closing the old one, and letting in-flight jobs finish on the old browser, keeps the swap invisible to workers. Keep a reference to the closing task in production code, as described in preventing task garbage collection with strong references.

Verify: over a day-long run, browser PSS shows a sawtooth reset at each recycle rather than a slow upward drift.

5. Shed load before the container is killed

Measure memory continuously and stop taking new jobs when it approaches the limit, so the process degrades instead of being killed mid-job:

class MemoryGate:
    def __init__(self, soft_limit_mib: float) -> None:
        self.soft = soft_limit_mib
        self.current = 0.0

    async def sample_forever(self) -> None:
        while True:
            self.current = await asyncio.to_thread(chromium_pss_mib)
            BROWSER_PSS.set(self.current)
            await asyncio.sleep(2)

    async def wait_for_room(self) -> None:
        while self.current > self.soft:
            await asyncio.sleep(0.5)                  # in-flight pages finish, memory falls

Workers call await gate.wait_for_room() before opening a context. A soft limit at 80% of the container's memory turns an unusually heavy batch of pages into a slowdown rather than an out-of-memory kill, which would lose every in-flight job at once. The same idea for request handling is load shedding when the event loop is overloaded.

Verify: a batch of deliberately heavy pages pauses the workers at the soft limit and the container is not OOM-killed.

What keeps this browser's memory bounded? A decision on How does memory grow with 4 outcomes. What keeps this browser's memory bounded? How does memory grow? with pages processed close pages/contexts per job 906 MiB -> flat 303-336 with concurrency size workers from per-page PSS ~20 MiB each here slowly over hours recycle the browser every N pages 145 ms spikes from heavy pages memory gate before new jobs slow down, not OOM Close what you open, size from measurements, and keep a safety valve.

Verification

Headless browser memory stays bounded when:

  • Browser memory is measured as PSS, sampled on a timer.
  • Pages (and preferably contexts) are closed after every job, so memory tracks concurrency, not volume.
  • The worker count fits the memory budget with headroom, from a measured per-page cost.
  • The browser is recycled on a schedule, and a memory gate sheds load before the limit.

Diagnostic Hook: plot browser PSS against pages processed and against open contexts on the same time axis. Memory that follows open contexts is healthy; memory that follows cumulative pages is a lifecycle bug — almost always a reused page or a context that is never closed.

Pitfalls & edge cases

  • Reusing pages across jobs. Measured: 906 MiB within 400 pages, and slower.
  • Summed RSS as the metric. It reported 321 MiB for a 174 MiB browser.
  • Sizing from average pages. Heavy pages need headroom; use a high percentile.
  • Recycling by closing first. Launch the replacement, then retire the old browser.

Frequently Asked Questions

Why does Playwright memory keep growing?

Usually because pages are reused across many navigations: reusing one page per worker grew Chromium to 906 MiB within 400 pages in testing, while a fresh context per page stayed between 303 and 336 MiB over 1,000 pages.

How much memory does each Playwright page use?

About 20 MiB of PSS per context with a loaded page on simple test pages, on top of about 174 MiB for the browser. Heavy real sites use much more; measure on your targets.

Should I restart the Playwright browser periodically?

It is cheap insurance against slow drift: closing and relaunching took 145 ms in testing. Launch the replacement first and close the old browser once its contexts finish.

How do I measure headless Chrome memory correctly?

Sum PSS (psutil memory_full_info().pss on Linux) over the browser's processes; summed RSS double-counts shared memory and reported nearly twice the PSS figure in testing.