Skip to content

Running Background Tasks in FastAPI Safely

"Return the response now, do the work afterwards" is one of the most useful things an async framework offers and one of the easiest to get quietly wrong. FastAPI's BackgroundTasks does it correctly for short work; asyncio.create_task() inside a handler looks equivalent and is not. Measured on a one-second background job with a shutdown triggered 100 ms in: the BackgroundTasks version kept the server alive until the job finished at 1.05 s, while the create_task version let the server exit at 0.29 s with the job still running — in a real process, killed. This guide covers what each mechanism guarantees, how to make an untracked task safe, and the point at which none of them is the right answer.

Prerequisites

Four ways to do work after the response A grid of 4 rows by 2 columns. Four ways to do work after the response approach survives a restart? bounded? BackgroundTasks no, but shutdown waits no: one task per request tracked create_task no, and shutdown may not wait no, unless you add a limit queue + worker tasks no yes: the queue is bounded external job queue yes: durable yes: the workers are Verified: the server waited for a BackgroundTask and exited while a bare task was still running.

1. Use BackgroundTasks for short, response-coupled work

BackgroundTasks runs after the response is sent and before the server considers the request finished:

from fastapi import BackgroundTasks, FastAPI


@app.post("/signup")
async def signup(email: str, background: BackgroundTasks):
    user = await create_user(email)
    background.add_task(send_welcome_email, user.id)    # runs after the response
    return {"id": user.id}

The response returned in 10.9 ms while the 300 ms task ran afterwards, and — crucially — the server's shutdown waited for it. That makes it right for work that is a continuation of the request and short enough that a deploy can wait: sending a notification, writing an audit record, invalidating a cache entry.

A synchronous function passed to add_task is run in the thread pool rather than on the loop, so it does not block other requests — measured, a request issued right after a 300 ms blocking background task was served in 0.6 ms. It does consume an executor thread, which the thread pool sizing rules still apply to.

What BackgroundTasks does not give you is durability or a bound. Each request adds its own task, so a burst of requests is a burst of concurrent background work with nothing limiting it.

Verify: the response returns before the task completes, and a shutdown waits for it.

2. Understand what a bare create_task loses

@app.post("/signup")
async def signup(email: str):
    asyncio.create_task(send_welcome_email(user.id))    # nothing owns this
    return {"id": user.id}

Three problems, in increasing order of how long they take to discover.

Shutdown does not wait. Verified above: the server exited at 0.29 s while the task had 700 ms left. In production that is a deploy silently discarding work.

Failures are invisible. An exception in an unobserved task produces this, on the loop's exception handler, at some arbitrary later time:

Task exception was never retrieved
future: <Task finished ... exception=ValueError('background failure nobody sees')>

It is not attached to a request, not in your structured logs, and not counted anywhere.

The task can be garbage collected. The event loop keeps only a weak reference, so the documented requirement is to hold a strong reference until the task completes. This one is genuinely hard to trigger — in testing, a task with no reference survived three explicit gc.collect() calls — but "hard to reproduce" is the worst category of bug to leave in a service.

Verify: grep for create_task( in request handlers; each one is a candidate for all three problems.

What a shutdown does to work in flight 4 lanes over time. What a shutdown does to work in flight BackgroundTasks running finished server (case 1) shutting down exits after create_task running killed mid-flight server (case 2) shutting down exits at 0.29 s time → Measured on a 1 s task: the server exited at 1.05 s and at 0.29 s respectively.

3. Track the tasks you create

When a task must outlive its request — because it is longer than a response should wait for — own it explicitly:

_background: set[asyncio.Task] = set()


def spawn(coro) -> asyncio.Task:
    task = asyncio.create_task(coro)
    _background.add(task)                              # strong reference
    task.add_done_callback(_background.discard)        # and released when done
    task.add_done_callback(_log_failure)
    return task


def _log_failure(task: asyncio.Task) -> None:
    if task.cancelled():
        return
    if (exc := task.exception()) is not None:
        logger.error("background task failed", exc_info=exc)

Then cancel the set during shutdown, in the lifespan:

    finally:
        for task in list(_background):
            task.cancel()
        await asyncio.gather(*_background, return_exceptions=True)

That fixes the reference, the invisible failures and the shutdown race in about fifteen lines. len(_background) is also a metric worth exporting — a number that only grows is a leak, as covered in tracking task growth in long-running services.

Verify: the set returns to its baseline size when the service is idle, and failures appear in the application log.

Making a fire-and-forget task safe 5 ordered steps. Making a fire-and-forget task safe task = create_task(work()) never bare tasks.add(task) a strong reference add_done_callback(discard) so the set does not grow log exceptions in the callback or they are silent cancel the set on shutdown in the lifespan finally Without the reference, the loop is the only owner; without the callback, failures vanish.

4. Bound the work, not just its lifetime

Tracking does not limit concurrency. A thousand requests spawning a thousand background HTTP calls will exhaust a connection pool, a rate limit or the database long before anything complains about task count. Put a queue in front:

queue: asyncio.Queue = asyncio.Queue(maxsize=1000)     # bounded on purpose


async def worker() -> None:
    while True:
        job = await queue.get()
        try:
            await handle(job)
        except Exception:
            logger.exception("job failed")             # never let the worker die
        finally:
            queue.task_done()

Worker tasks are started in the lifespan and cancelled with everything else, and the handler becomes queue.put_nowait(job) with a QueueFull branch that returns 503 — real back-pressure, rather than accepting work you cannot do. Building a worker pool with TaskGroup covers the structure in detail.

Verify: under a burst, the queue reaches its bound and requests are rejected rather than the service degrading.

5. Know when it belongs outside the process

Every in-process option shares one property: a restart loses the work. Deploys, autoscaling, node evictions and crashes are all routine, so the real question is what losing the work costs.

  • Losing it is fine — cache warms, best-effort notifications, metrics. BackgroundTasks or a tracked task is right.
  • Losing it is noticeable but recoverable — a search index update that a nightly job would fix. A bounded in-process queue is a reasonable trade.
  • A customer notices — payment capture, order fulfilment, email that must be sent. It belongs in a durable queue: a Postgres job table, a broker, or a job framework such as arq or taskiq.

The transition point is worth naming precisely, because teams usually cross it without noticing: as soon as someone asks "did that email actually go out?", the work needed a durable record, and no amount of task tracking provides one.

Verify: for each background job, answer what a restart mid-execution costs — and check the mechanism matches.

Where does this work belong? A decision on What happens if this work is lost with 3 outcomes. Where does this work belong? What happens if this work is lost? nothing important BackgroundTasks emails, cache warms noticeable but survivable queue + workers bounded, in process a customer notices a durable job queue Postgres, Redis, a broker The question is not how long the work takes; it is what a restart is allowed to destroy.

Verification

Background work is handled safely when:

  • Short response-coupled work uses BackgroundTasks, which shutdown waits for.
  • Every other task is tracked in a set with a done callback.
  • Failures are logged with the exception, not left to the loop's handler.
  • Concurrency is bounded by a queue or semaphore, not by request rate.
  • Shutdown cancels and awaits the tracked tasks.
  • Work that must not be lost is durable, stored before the response returns.

Pitfalls & edge cases

  • asyncio.create_task with no reference. The loop holds only a weak one; keep a strong reference until completion.
  • Long-running work in BackgroundTasks. Every deploy waits for it, or timeout_graceful_shutdown kills it mid-way.
  • A worker task that dies on an exception. Wrap the body in try/except so one bad job does not stop the pool.
  • Background work holding request state. The request object and its connection are gone after the response; copy what you need.
  • Per-worker queues. With several uvicorn workers, each has its own queue and its own depth; the metric is per process.
  • Blocking calls in background tasks. They consume executor threads shared with to_thread; give heavy work its own executor.

Frequently Asked Questions

What is the difference between BackgroundTasks and asyncio.create_task in FastAPI?

BackgroundTasks is owned by the framework: the task runs after the response and the server's shutdown waits for it. A bare create_task is owned by nothing — measured, the server exited at 0.29 s while a one-second task was still running, and its exceptions surface only as "Task exception was never retrieved".

Why does my FastAPI background task not finish?

Most likely the process shut down first. Only BackgroundTasks keeps the server alive; tasks created with create_task are abandoned when the worker exits. Track them in a set and cancel-and-await them in the lifespan, or move the work to a durable queue.

How do I run a background task that outlives the request in FastAPI?

Create it with asyncio.create_task, keep a strong reference in a module-level set, discard it in a done callback, and log exceptions there. Cancel and await the whole set in the lifespan teardown so shutdown is orderly.

Do FastAPI background tasks block other requests?

An async background task shares the event loop, so it competes with request handling but does not block it unless it does blocking work. A synchronous function passed to add_task is run in the thread pool: a request issued right after a 300 ms blocking background task was still served in 0.6 ms.

When should background work move to Celery or a job queue?

As soon as losing the work on a restart would be noticed. In-process tasks die with the process, so anything a customer would ask about — payments, fulfilment, transactional email — needs a durable record written before the response returns.