Skip to content

Reducing Pickle Overhead in ProcessPoolExecutor Payloads

Moving CPU work to a ProcessPoolExecutor sometimes makes a service slower, and the reason is almost always the same: every argument is pickled in the parent, written through a pipe, and unpickled in the child, then the result makes the same journey back. For a 200 MB DataFrame and 40 ms of computation, the serialisation is the workload — you have built an expensive way to copy memory. The event loop feels it too, because pickling happens on the calling thread, so the "offload" blocks the loop for exactly as long as the copy takes.

The fix is not to abandon process pools but to change what crosses the boundary. This guide measures the real cost first, then applies the four techniques that matter: send references instead of payloads, batch small items so per-call overhead amortises, use shared memory for large arrays, and recognise the cases where a thread (or no offload at all) is simply the better tool.

Prerequisites

  • Python 3.11+; standard library plus optionally NumPy for the shared-memory example:
pip install numpy
Round trip for a 182 MB payload, 40 ms of work Four bars comparing serialisation cost with compute time. Round trip for a 182 MB payload, 40 ms of work pickle.dumps in the parent 310 ms — on the loop thread pipe write + read 95 ms pickle.loads in the child 241 ms the actual computation 40 ms Measured on one host with protocol 5; times scale with payload size, not worker count. When transfer exceeds a third of the compute, more workers cannot help.

1. Measure the transfer, not just the work

Before optimising, find out what fraction of the round trip is serialisation.

import pickle
import time


def transfer_cost(obj) -> None:
    t0 = time.perf_counter()
    blob = pickle.dumps(obj, protocol=pickle.HIGHEST_PROTOCOL)
    t1 = time.perf_counter()
    pickle.loads(blob)
    t2 = time.perf_counter()
    print(f"{len(blob) / 1e6:8.2f} MB   dump {(t1 - t0) * 1e3:7.1f} ms   "
          f"load {(t2 - t1) * 1e3:7.1f} ms")


transfer_cost(rows)          # e.g. 182.40 MB   dump  310.2 ms   load  240.7 ms

Compare that total against the work the child will do. A useful rule: if dump + load exceeds roughly a third of the computation time, the boundary is your bottleneck and no amount of extra workers will help — you are queueing on a pipe, not on CPU. Also note that both halves of the copy run on your thread, so with an event loop involved they are a loop stall, not background cost. Verify: the measured dump time matches the loop-lag spike you see when submitting the job.

2. Send a reference, not the data

The cheapest payload is one the child can fetch or rebuild itself.

import asyncio
from concurrent.futures import ProcessPoolExecutor
from pathlib import Path

POOL = ProcessPoolExecutor(max_workers=4)


def summarise_file(path: str, offset: int, length: int) -> dict:
    """Runs in the child: reads its own slice, returns a small result."""
    with open(path, "rb") as fh:
        fh.seek(offset)
        chunk = fh.read(length)
    return compute_summary(chunk)                 # a few hundred bytes back


async def summarise(path: Path, chunks: list[tuple[int, int]]) -> list[dict]:
    loop = asyncio.get_running_loop()
    return await asyncio.gather(*(
        loop.run_in_executor(POOL, summarise_file, str(path), off, length)
        for off, length in chunks))

Three strings and two integers cross the boundary instead of hundreds of megabytes. The same idea covers a database row id instead of the row, an S3 key instead of the object, and a memory-mapped file path instead of its contents. It only works when the child can reach the source, which is the usual case for files, object storage and databases. Verify: payload size per call drops to bytes, and loop lag during submission returns to baseline.

3. Batch small items

Per-call overhead — pickling, pipe write, scheduling — is roughly constant, so tiny tasks are dominated by it.

import asyncio
from concurrent.futures import ProcessPoolExecutor

POOL = ProcessPoolExecutor(max_workers=8)


def process_batch(items: list[dict]) -> list[dict]:
    return [transform(item) for item in items]    # one hop, many items


async def process_all(items: list[dict], batch_size: int = 500) -> list[dict]:
    loop = asyncio.get_running_loop()
    batches = [items[i:i + batch_size] for i in range(0, len(items), batch_size)]
    results = await asyncio.gather(*(
        loop.run_in_executor(POOL, process_batch, batch) for batch in batches))
    return [row for batch in results for row in batch]

Submitting 100,000 items individually means 100,000 pickles and 100,000 scheduling hops, which typically costs more than the transformation. Batching to 500 reduces that to 200 hops. Size the batch so each one takes tens to hundreds of milliseconds in the child: shorter and overhead dominates again, much longer and you lose the ability to balance load across workers. Executor.map(..., chunksize=N) does the same thing for simple cases. Verify: total throughput improves roughly in proportion to the reduction in call count, up to the point where batches take ~100 ms each.

Where per-call overhead goes A left-to-right comparison of unbatched and batched submission counts. Where per-call overhead goes 100k items one call each 100k pickles plus 100k hops batch by 500 200 calls 200 pickles overhead amortised Batch until each call takes roughly 100 ms in the child, then stop.

4. Use shared memory for large arrays

For numeric data, multiprocessing.shared_memory removes the copy entirely.

import asyncio
import numpy as np
from concurrent.futures import ProcessPoolExecutor
from multiprocessing import shared_memory

POOL = ProcessPoolExecutor(max_workers=4)


def analyse_shm(name: str, shape: tuple, dtype: str, start: int, stop: int) -> float:
    shm = shared_memory.SharedMemory(name=name)          # maps, does not copy
    try:
        array = np.ndarray(shape, dtype=np.dtype(dtype), buffer=shm.buf)
        return float(array[start:stop].std())
    finally:
        shm.close()                                       # unmap in the child


async def analyse(array: np.ndarray, workers: int = 4) -> list[float]:
    shm = shared_memory.SharedMemory(create=True, size=array.nbytes)
    try:
        shared = np.ndarray(array.shape, dtype=array.dtype, buffer=shm.buf)
        shared[:] = array[:]                              # one copy, then none
        loop = asyncio.get_running_loop()
        step = len(array) // workers
        return await asyncio.gather(*(
            loop.run_in_executor(POOL, analyse_shm, shm.name, array.shape,
                                 array.dtype.str, i * step, (i + 1) * step)
            for i in range(workers)))
    finally:
        shm.close()
        shm.unlink()                                      # the creator must unlink

Only the block's name, shape and dtype are pickled — a few dozen bytes regardless of array size. The rules to respect: exactly one process creates and unlinks, every other process closes without unlinking, and concurrent writers need their own disjoint slices or explicit synchronisation. Forgetting unlink() leaks a block in /dev/shm that survives the process. Verify: ls /dev/shm is clean after the run, and submission cost is independent of array size.

5. Know when a process pool is the wrong tool

Three cases where the boundary can never pay for itself.

# (a) the library already releases the GIL -> threads, no serialisation at all
result = await asyncio.to_thread(np.linalg.svd, matrix)      # NumPy releases the GIL

# (b) the work is shorter than the transfer -> do it inline
digest = hashlib.sha256(small_blob).hexdigest()              # microseconds

# (c) the data is huge and the result is huge -> move the code, not the data
await run_query_in_the_database(sql)                          # aggregate at the source

NumPy, SciPy, Pillow, zlib, hashlib and most compiled extensions release the GIL during their heavy work, so a thread gives real parallelism with zero serialisation — always check before reaching for processes. When the computation is measured in microseconds, any offload is a loss. And when both input and output are large, the honest fix is usually to push the computation to where the data already lives. Verify: benchmark the inline, thread and process variants on real data; the winner is frequently not the one you assumed.

Should this work cross a process boundary? A decision diamond on GIL behaviour with three outcome cards. Should this work cross a process boundary? Does the library release the GIL? yes — NumPy, zlib, Pillow use a thread real parallelism, zero copy no, and the data is small use a process pool batch and send references no, and both sides are huge move the code to the data aggregate at the source The winner is frequently not the one the intuition picks; benchmark all three.

Verification

  • Transfer is a minority of the round trip. Measured dump plus load stays well under a third of the child's compute time.
  • Loop lag is flat during submission. No p99 spike correlated with job submission, which would indicate pickling on the loop thread.
  • Throughput scales with workers. Adding workers increases throughput near-linearly until CPU saturates, rather than plateauing on pipe bandwidth.

Pitfalls and edge cases

  • Pickling on the loop thread. run_in_executor serialises arguments in the caller, so a large payload stalls the loop before any parallelism starts.
  • Unpicklable arguments. Lambdas, local functions, open connections and locks raise at submission; pass module-level callables and plain data.
  • A leaked shared-memory block. Without unlink(), the segment survives in /dev/shm until reboot.
  • Batches that are too large. Load balancing suffers and one slow batch delays the whole result set.
  • Re-sending unchanged reference data every call. Load it once per worker with an initialiser instead.
  • Assuming processes beat threads for numeric work. Most numeric libraries release the GIL; threads then win outright.
  • Forgetting fork/spawn differences. On spawn platforms (macOS, Windows), every argument and the module are re-imported per worker, making startup and pickling costs higher.

Frequently Asked Questions

Why is my ProcessPoolExecutor slower than running the work inline?

Because serialisation dominates. Every argument is pickled in the parent, copied through a pipe and unpickled in the child, then the result returns the same way. Measure dump plus load time against the child's compute time: if it exceeds roughly a third, the boundary is the bottleneck and more workers will not help.

How do I avoid pickling large data for a process pool?

Send a reference the child can resolve itself — a file path with an offset, a database key, an object-storage key — or place numeric arrays in multiprocessing.shared_memory and pass only the block name, shape and dtype. Both reduce the payload to bytes regardless of data size.

What batch size should I use for process pool tasks?

Aim for batches that take tens to low hundreds of milliseconds in the child. Smaller and per-call overhead dominates; much larger and load balancing suffers because one slow batch holds up the result. For simple mapping, Executor.map with an explicit chunksize achieves the same effect.

Does pickling block the event loop?

Yes. run_in_executor serialises the arguments on the calling thread before the work is dispatched, so submitting a large payload stalls the loop for the duration of the dump — often the very spike people are trying to remove. Shrinking the payload is what makes the offload actually asynchronous.