Skip to content

Choosing Between Celery, arq and taskiq

The choice between Python's task queues is usually framed as a feature comparison, and for an async service the more useful framing is: does the worker speak your code's language? Celery's workers run synchronous functions in processes, so every coroutine needs a bridge; arq and taskiq run async def tasks directly on an event loop, so a task is just a coroutine with pools that already exist. That difference shows up in throughput on I/O-bound work — 200 jobs each awaiting 10 ms took 0.63 s on four Celery prefork processes, 0.32 s on one arq worker and 0.21 s on one taskiq worker — and it shows up in how much code you maintain around the queue.

Prerequisites

Draining 200 I/O-bound jobs 3 bars comparing Celery, 4 prefork processes with the others. Draining 200 I/O-bound jobs Celery, 4 prefork processes 0.63 s arq, 1 worker, 20 concurrent 0.32 s taskiq, 1 worker, 20 concurrent 0.21 s Each job awaits 10 ms; Redis as the broker in all three cases, on one machine. One async worker beat four processes here because the work was waiting, not computing.

1. Compare the execution models, not the feature lists

Celery runs a pool of worker processes, each executing one synchronous task at a time. Concurrency equals the process count, so 200 tasks that each wait 10 ms on four processes take at least 200 / 4 × 10 ms. Measured: 0.63 s end to end, with enqueueing itself taking 0.10 s.

arq runs one event loop per worker with a configurable number of concurrent jobs (max_jobs). The same 200 jobs, one worker, 20 concurrent: 0.32 s, with 0.05 s to enqueue.

taskiq has the same shape — one loop, max_async_tasks concurrent — and completed in 0.21 s, with 0.03 s to enqueue.

The numbers are not a benchmark of "which library is fastest"; they are a demonstration of the model. Awaiting work does not need a process each, so an async worker with twenty concurrent jobs uses one core where Celery uses four. Invert the workload — tasks that compute rather than wait — and Celery's processes win, because twenty coroutines on one loop still share one core.

Verify: measure your own tasks' wait-to-compute ratio; it decides which model fits.

2. Know that retry semantics differ, in both directions

This is the single most surprising difference in practice. In arq, a task that raises a plain exception is not retried:

a plain exception is NOT retried: job failed with ValueError('plain exception')
  failed job job_try: 1

Retries are explicit:

from arq.worker import Retry


async def flaky(ctx):
    if not await attempt():
        raise Retry(defer=ctx["job_try"] * 5)          # re-queue, with backoff

Verified: the same task retried to ok on try 3 with Retry, and gave up at max_tries. In taskiq, retries are opt-in per task via middleware:

@broker.task(retry_on_error=True, max_retries=3)
async def flaky(n: int) -> str:
    ...

And Celery retries only when the task calls self.retry() or declares autoretry_for. Three libraries, three defaults — none of which is "retry on any exception". Read the rule before relying on it, because "the queue will retry it" is an assumption that fails silently.

Verify: write a deliberately failing task and count the attempts your configuration actually produces.

Retry behaviour is not the same 2 columns contrasting arq, taskiq. Retry behaviour is not the same arq explicit retries a plain exception fails the job verified: job_try stayed 1 raise Retry(defer=...) to re-queue max_tries caps the loop taskiq opt-in per task retry_on_error=True on the decorator max_retries bounds it middleware controls the policy result backend keeps the error Read your framework’s retry rule before relying on it: the defaults differ in both directions.

3. Weigh the features you will actually use

Celery's advantages are real and specific:

  • Routing and priorities. Multiple queues, exchanges, per-task routing, priority levels. arq has one list; taskiq has a small set of broker implementations.
  • Scheduling. celery beat with database-backed schedules, and an ecosystem around it. Both async queues handle cron entries in the worker itself, which is simpler and less flexible — see scheduling cron jobs inside an asyncio service.
  • Ecosystem. Flower for monitoring, Django integration, and fifteen years of answered questions. For an unusual problem, someone has hit it before.
  • Brokers. Redis, RabbitMQ, SQS and more; the async queues are Redis-centric.

Against that, the async-native queues offer codebases small enough to read in an afternoon, no bridge, no prefork subtleties, and a worker whose memory profile is one process rather than N. For a service whose background work is "call three APIs and write a row", that is the better trade.

Verify: list the Celery features you use; if it is "delay a function", the async queues cover it.

What you get, and what you give up A grid of 5 rows by 2 columns. What you get, and what you give up Celery arq / taskiq async tasks via a bridge you write native: async def tasks scheduling beat, with many backends cron entries in the worker routing and priorities rich: queues, exchanges basic: one list, or streams ecosystem Flower, Django, years of answers small, readable codebases brokers Redis, RabbitMQ, SQS and more Redis (taskiq adds a few) Celery buys features and community; the async-native queues buy simplicity.

4. Use the primitives each one gives you

Both async queues have useful features worth knowing before choosing.

arq's _job_id gives deduplication for free — enqueueing the same id twice returns None the second time:

await redis.enqueue_job("rebuild_index", tenant_id, _job_id=f"rebuild:{tenant_id}")

Verified. That turns "coalesce repeated requests for the same work" into one argument, where Celery needs a lock or a separate dedup layer. _defer_by and _defer_until schedule a job for later, and the job's result and status are readable by id afterwards.

taskiq keeps full type information on task calls — await add.kiq(1, 2) is type-checked against the function's signature — and separates brokers from result backends, so results can go somewhere other than the queue. Its middleware system is the extension point: retries, metrics and tracing are all middleware rather than framework internals.

Verify: the feature you need exists before committing, rather than after.

5. Consider whether the queue should be the database

All three queues are separate systems with their own durability story, and for work that is about database state, a job table is often the better answer:

SELECT id, payload FROM jobs
 WHERE status = 'pending' AND run_at <= now()
 ORDER BY run_at LIMIT 10
 FOR UPDATE SKIP LOCKED

The job and the data it operates on then share one transaction, so a job cannot be enqueued for a row that was rolled back — the same property that makes the transactional outbox work. There is no extra broker to run, and the whole state is queryable with SQL, which matters more in an incident than any dashboard.

What you give up is throughput measured in thousands per second, fan-out to many consumers, and the scheduling features. Building a durable job queue on Postgres covers the implementation; the rule of thumb is that under a few hundred jobs per second, the database is usually enough.

Verify: estimate your peak job rate; if it is in the tens per second, a job table is a serious option.

Which queue for this service? A decision on What is already true with 3 outcomes. Which queue for this service? What is already true? Celery is already running keep it, bridge async one module, one helper new service, async code arq or taskiq no bridge, less to run work must survive anything a database queue your own transactions The durable-queue answer is the one to reach for when the job IS the database write.

Verification

The choice is sound when:

  • The execution model matches the work: processes for CPU, one loop for I/O.
  • Retry behaviour is verified, not assumed from the documentation's tone.
  • The features you actually use are listed and present.
  • The operational cost is counted: brokers, dashboards, result backends.
  • Durability requirements are met, with a database queue where the job is a database change.
  • Migration cost is understood if the existing queue is Celery.

Pitfalls & edge cases

  • Assuming failed tasks retry. All three defaults differ; none retries every exception automatically.
  • Celery with asyncio.run everywhere. Works, but pays connection setup per task — measured at 71x for a trivial query.
  • arq's single queue. Priority means running separate workers on separate queue names.
  • Result backends left on by default. Storing every result in Redis grows unbounded; set expiry or disable it.
  • Prefork memory. Celery's worker count multiplies memory and pool sizes; eight workers with max_size=8 is 64 connections.
  • Choosing on benchmarks alone. The numbers here reflect one workload shape; yours decides the answer.

Frequently Asked Questions

Is arq or taskiq faster than Celery?

For I/O-bound work, in these measurements yes: 200 jobs each awaiting 10 ms took 0.63 s on four Celery prefork processes, 0.32 s on one arq worker and 0.21 s on one taskiq worker. That is the execution model, not the code quality — for CPU-bound tasks Celery's processes use more cores.

Can Celery run async tasks natively?

Not as coroutine task functions. Tasks are synchronous, so async code needs a bridge — asyncio.run per task, or one event loop per worker process with run_coroutine_threadsafe. The bridge works well but is machinery you maintain.

Does arq retry failed jobs automatically?

No. A task that raises a plain exception is marked failed with job_try 1, verified. Retries require raising arq.worker.Retry, optionally with a defer, and are capped by max_tries. taskiq opts in per task with retry_on_error, and Celery needs self.retry() or autoretry_for.

When should I use a database table instead of a task queue?

When the job is about database state and must not be lost — the job row and the data change then share one transaction. It also means no extra broker to operate and a queue you can inspect with SQL. Below a few hundred jobs per second it is usually enough.

Should I migrate an existing Celery deployment to arq or taskiq?

Only with a reason: most async tasks, a bridge that has become a maintenance burden, or a wish to shed operational weight. Celery's routing, scheduling and ecosystem are genuine advantages, and a migration touches every producer as well as every worker.