Skip to content

Scheduling Cron Jobs Inside an asyncio Service

A periodic task inside a service starts as four lines — while True: await work(); await asyncio.sleep(60) — and every one of the four assumptions in it is wrong in production. The schedule drifts by the duration of the work on every iteration. Every instance of the service runs the job, so three replicas send three emails. A restart at the wrong moment skips a run entirely, or repeats one. And a run that takes longer than the interval overlaps with the next. This guide fixes each in turn, with the drift and locking behaviour measured rather than argued.

Prerequisites

  • Python 3.11+; croniter for cron expressions and redis for the cross-instance lock.
  • Task ownership from Task Scheduling & Lifecycle.
  • Timezone handling via zoneinfo, which is standard library.
Ten ticks of a 100 ms schedule, with 20 ms of work 2 bars comparing sleep(period) after the work with the others. Ten ticks of a 100 ms schedule, with 20 ms of work sleep(period) after the work 1.20 s (+0.20 drift) sleep until the next slot 1.02 s (+0.02 drift) The nominal schedule is 1.00 s of wall clock; the work itself takes 20 ms per tick. Drift compounds: the first shape loses the duration of the work on every single tick.

1. Sleep until the next slot, not for the period

The naive loop sleeps after doing the work, so each iteration is period + work. Over ten ticks of a 100 ms schedule with 20 ms of work:

sleep(period) inside the loop: 1.20s elapsed (drifted +0.20s)
sleep until the next slot:     1.02s elapsed (drifted +0.02s)

A 20% drift after ten iterations, compounding indefinitely — the "hourly" job runs at 09:00, 10:04, 11:08. The fix is to compute the next fire time and sleep until it:

loop = asyncio.get_running_loop()
next_slot = loop.time() + period
while True:
    await asyncio.sleep(max(0.0, next_slot - loop.time()))
    await run_job()
    next_slot += period                                # from the slot, never from now()

next_slot += period rather than next_slot = loop.time() + period is the whole trick: the schedule is defined by the slots, and a slow run consumes its own slot rather than pushing every later one back. Running periodic tasks without drift covers the interval case in more depth; the cron case is the same idea with a calendar.

Verify: after an hour, the job's start times are still aligned with the intended schedule.

2. Use cron expressions for calendar schedules

Intervals answer "every N seconds"; business schedules are calendar-shaped — 02:00 daily, Monday mornings, the first of the month. croniter turns an expression and a starting point into the next datetimes:

from croniter import croniter

def next_fire(expression: str, after: datetime, tz: ZoneInfo) -> datetime:
    return croniter(expression, after.astimezone(tz)).get_next(datetime)

Use timezone-aware datetimes throughout, and pick the timezone deliberately: UTC for anything technical, a business timezone only when humans care that the report arrives at 9 a.m. their time.

That choice has a consequence worth seeing before it surprises you. A daily 01:30 job in Europe/London across the spring transition:

2026-03-28T01:30:00+00:00
2026-03-29T02:00:00+01:00     <- 01:30 did not exist that day
2026-03-30T01:30:00+01:00

The clock skipped 01:00–02:00, so the run moved to 02:00. In autumn the reverse happens and a local hour repeats, so a naive scheduler can fire twice. Scheduling in UTC avoids both; scheduling in a local timezone means accepting them and making the job idempotent.

Verify: print the next several fire times across a DST boundary before deploying a local-time schedule.

3. Elect a single runner across instances

Three replicas each run their own scheduler, so each slot fires three times. A lock keyed by the slot — not by the job — makes the run exactly-once per slot:

async def claim(redis, job: str, slot: datetime, ttl: int = 300) -> bool:
    key = f"cron:{job}:{slot.isoformat()}"
    return bool(await redis.set(key, INSTANCE_ID, nx=True, ex=ttl))

Verified with five instances racing for one slot: exactly one acquired it, and the key carried a 30-second TTL so a holder that dies does not block the next slot forever.

Keying by slot rather than by job matters. A job-keyed lock held by a crashed instance blocks every future run until its TTL expires; a slot-keyed lock affects only that slot, and the next one starts clean. Set the TTL comfortably above the job's normal duration and below the schedule interval.

For work that must not be missed even if the winning instance dies mid-run, this is not enough on its own — pair it with a durable job row whose state survives the process.

Verify: with several replicas running, the job's own logs show one execution per slot.

The four things a scheduler in a service must decide A grid of 4 rows by 2 columns. The four things a scheduler in a service must decide question wrong answer right answer drift sleep for the period sleep until the next slot many instances let all of them run it one lock per slot, with a TTL after downtime silently skip decide: catch up, or run once a slow run start the next anyway skip, or queue, deliberately Verified: 5 instances racing for one slot, exactly 1 acquired the lock.

4. Decide what happens after downtime

A service that was down from 09:00 to 14:00 has an hourly job with five missed runs:

after 5 hours of downtime, hourly schedule missed 5 runs: ['10:00', '11:00', '12:00', '13:00', '14:00']

There are three defensible policies, and the wrong thing is to have none:

  • Skip. Resume from the next slot. Right for anything whose value is "the current state" — a cache refresh, a metrics scrape.
  • Run once. Execute a single catch-up run immediately. Right for idempotent aggregations that recompute from source data anyway.
  • Catch up all. Run every missed slot in order. Right when each run produces a distinct artefact — a daily report per date, an hourly export.

Implementing any of them needs the last successful run persisted somewhere that survives a restart:

last = await store.get_last_run(job) or datetime.now(tz)
missed = list(iterate_slots(expression, after=last, until=now))
for slot in missed[-MAX_CATCHUP:]:                     # a cap, always
    await run_job(slot)

The cap stops a service that was down for a week from starting 168 jobs at once.

Verify: stop the service across several slots, restart it, and confirm the chosen policy is what happens.

The scheduler loop 5 ordered steps. The scheduler loop compute the next slot croniter from the last slot sleep until that instant not for a fixed period take the lock for the slot SET NX EX, keyed by slot run it as a tracked task with a timeout advance from the slot time never from now() Advancing from the slot rather than from now() is what stops the schedule sliding.

5. Handle overlap and failures explicitly

A run that outlives its interval raises a question the code must answer:

if job_task is not None and not job_task.done():
    SKIPPED.inc()                                      # or queue it, or let them overlap
    return
job_task = asyncio.create_task(run_with_timeout(slot))

Skipping is the safe default for most jobs and should be counted — a skip metric that climbs is telling you the job no longer fits its schedule. Always bound the run itself, or one stuck execution silently suspends the schedule forever:

async def run_with_timeout(slot):
    try:
        async with asyncio.timeout(MAX_RUN_SECONDS):
            await job(slot)
    except TimeoutError:
        log.error("job %s exceeded its budget for slot %s", name, slot)
    except Exception:
        log.exception("job %s failed for slot %s", name, slot)   # never kill the scheduler

The scheduler loop must survive a failing job. An unhandled exception inside it ends the loop, and the symptom — "the nightly report stopped running three weeks ago" — takes weeks to notice. The scheduler task itself belongs to the lifespan, cancelled on shutdown like any other background task.

Verify: a job that raises leaves the scheduler running, and the next slot fires normally.

Where should this scheduled job live? A decision on What does the job need with 3 outcomes. Where should this scheduled job live? What does the job need? the service’s own state in the service with a lock per slot to fan out to workers a scheduler that enqueues beat, or a small ticker isolation and its own image a platform CronJob no scheduler code at all A job that only enqueues work is a good fit for a platform cron: nothing is lost if it is late.

Verification

An in-service scheduler is correct when:

  • Slots do not drift: start times stay aligned over hours.
  • One instance runs each slot, enforced by a slot-keyed lock with a TTL.
  • Downtime has a policy, with a cap on catch-up runs.
  • Overlap is decided and counted, not accidental.
  • Every run is bounded by a timeout, and failures never end the loop.
  • Schedules are timezone-explicit, with DST behaviour checked.

Pitfalls & edge cases

  • sleep(interval) after the work. Drifts by the work's duration every iteration — 20% in the measurement above.
  • Naive datetimes. datetime.now() without a timezone silently uses the host's, which differs between your laptop and the cluster.
  • Job-keyed locks. A crashed holder blocks every future run until the TTL expires; key by slot.
  • Unbounded catch-up. A week of downtime starts a week of jobs simultaneously.
  • A scheduler that dies quietly. Wrap the job call, not the loop, and alert on "no run since".
  • Assuming platform cron is exact. Kubernetes CronJob can start late and may skip under load; the job still needs idempotence.

Frequently Asked Questions

How do I run a periodic task in an asyncio service without drift?

Track the next fire time and sleep until it, advancing the target by the period each iteration rather than sleeping for the period after the work. Measured over ten ticks of a 100 ms schedule with 20 ms of work, the drifting version was 0.20 s late and the slot-based one 0.02 s.

How do I stop every replica running the same scheduled job?

Take a lock keyed by the job and the slot before running: SET key NX EX in Redis. Verified with five instances racing for one slot, exactly one acquired it. Key by slot so a crashed holder does not block future runs, and set the TTL above the job's normal duration.

What should happen to scheduled jobs missed while a service was down?

Pick a policy: skip to the next slot for state-refresh jobs, run once for idempotent recomputation, or replay every missed slot when each produces a distinct artefact. Persist the last successful run so the gap is known, and cap the number of catch-up runs.

Do cron schedules handle daylight saving time correctly?

They handle it consistently, not intuitively. A daily 01:30 job in Europe/London moved to 02:00 on the spring-forward day, because 01:30 did not exist; in autumn a local hour repeats. Schedule in UTC for technical jobs, and make local-time jobs idempotent.

Should scheduling live in the service or in the platform?

In the service when the job needs its state and connections, with a lock for single execution. In the platform — a Kubernetes CronJob or systemd timer — when the job is self-contained and being a few minutes late is acceptable. A job that only enqueues work is a good fit for platform cron.