Background Jobs, Scheduling, and Worker Pools

Introduction

An HTTP request should not wait for a report export, a batch email, or a resize operation that takes several seconds. Background jobs move work that can finish later out of the request path. Persist enough information to run each job, and let workers process it independently.

That sounds simple until a worker crashes mid-job, a retry sends the same email twice, or a burst fills memory. This guide walks through the job lifecycle, a bounded worker pool, recurring schedules, and the limits needed to keep background work predictable.

Job lifecycle and worker pool

A job should move through a small set of explicit states: queued, running, succeeded, or failed. A scheduler creates jobs for recurring work; it should not perform the work itself. Workers claim jobs, execute them, and record an outcome. On a crash, a lease or visibility timeout lets another worker reclaim work that was left running.

flowchart LR
    A[Request or schedule] --> B[Persist job]
    B --> C[Bounded queue]
    C --> D[Worker claims job]
    D --> E[Run idempotent handler]
    E --> F{Outcome}
    F -->|Success| G[Mark complete]
    F -->|Retryable failure| H[Retry with delay]
    F -->|Permanent failure or limit reached| I[Dead-letter review]

The queue is the boundary between accepting work and doing it. A bounded queue gives the service a way to say “not now” before memory or downstream services are overwhelmed. The worker pool controls how many jobs can run at once. Increasing the worker count is useful only while the database, API, or CPU has capacity for the extra load.

A small bounded worker pool

This in-process Python example shows the shape of a pool. It is useful for understanding concurrency and shutdown, but an in-memory queue loses pending work if the process exits. Use a durable broker or database-backed queue when jobs must survive deployments or machine failures.

import asyncio
from collections.abc import Awaitable, Callable
from dataclasses import dataclass


@dataclass(frozen=True)
class Job:
    job_id: str
    kind: str
    payload: dict[str, str]


Handler = Callable[[Job], Awaitable[None]]


async def worker(
    queue: asyncio.Queue[Job | None],
    handlers: dict[str, Handler],
) -> None:
    while True:
        job = await queue.get()
        try:
            if job is None:
                return
            handler = handlers[job.kind]
            await handler(job)
        except Exception as exc:
            # A durable queue should record the error and apply its retry policy.
            print(f"job failed: {job.job_id}: {exc}")
        finally:
            queue.task_done()


async def run_pool(
    jobs: list[Job], handlers: dict[str, Handler], worker_count: int = 4
) -> None:
    queue: asyncio.Queue[Job | None] = asyncio.Queue(maxsize=100)
    workers = [asyncio.create_task(worker(queue, handlers))
               for _ in range(worker_count)]

    for job in jobs:
        await queue.put(job)  # Waits when the bounded queue is full.

    await queue.join()
    for _ in workers:
        await queue.put(None)
    await asyncio.gather(*workers)

In a real service, the queue adapter should own persistence, claiming, acknowledgements, and retry scheduling. Keep handlers focused on the job’s business operation. Pass dependencies such as database clients into handlers at application startup rather than constructing a new connection for each job.

Make retries safe

Most queues provide at-least-once delivery: a job can run again if a worker finishes the side effect but crashes before acknowledging the message. Give each job a stable ID, and make the handler idempotent where practical. For example, use a unique constraint on payment_refund_id before issuing a refund, or store a completion record in the same transaction as a database update.

Retry only errors that may clear on their own. Use exponential backoff with jitter, cap the attempt count, and send exhausted jobs to a dead-letter queue for inspection. A permanent validation error will not improve after ten retries; retrying it just burns worker time.

Scheduling recurring work

A scheduler should enqueue a job with a scheduled timestamp and a unique occurrence key. If two scheduler instances wake at the same time, a database uniqueness constraint or distributed lock should prevent duplicate creation. Store the intended schedule in a consistent timezone, usually UTC, and define what happens when the service was down at the scheduled time: skip the run, enqueue one catch-up job, or replay every missed interval.

Avoid treating a long-running worker’s local timer as the source of truth. Process restarts reset in-memory timers, and multiple replicas may all run the same schedule. Use a scheduler with durable state or a database-backed schedule table, then keep the scheduled action itself idempotent.

Trade-offs and production failures

Queue growth and overload

If arrival rate stays above completion rate, queue age rises even when workers appear healthy. Cap queue depth, reject or defer new work when the cap is reached, and expose the rejection to the caller. Scale workers only after checking whether the downstream dependency can handle the added concurrency. This is the same producer-consumer problem described in the project’s backpressure guide.

Worker crashes and stuck jobs

A process can stop after claiming a job but before acknowledging it. Use a visibility timeout or lease that expires, heartbeat long jobs, and reclaim jobs whose lease has expired. Make the lease longer than normal execution time, but do not rely on it to prevent duplicate execution; idempotency still matters.

Retry storms and poison jobs

When a dependency fails, immediately retrying every queued job can overwhelm it during recovery. Add exponential backoff and jitter, cap retries, and pause a queue or job type when errors spike. Quarantine malformed or repeatedly failing jobs with the error, attempt count, and relevant correlation ID so an operator can decide whether to repair or discard them.

Shutdown during deployment

Workers need a graceful shutdown path. Stop claiming new jobs, allow current jobs a bounded drain period, then leave unfinished work unacknowledged so it can be reclaimed. Keep job handlers idempotent because a forced termination can happen at any point.

Observability and security

Track queue depth and oldest-job age by job type, completed and failed counts, retry counts, execution duration, and time spent waiting before execution. Alert on sustained queue age and rising dead-letter volume; queue depth alone can look fine while a small number of old jobs are stuck. Include job ID and request or schedule correlation ID in structured logs, but avoid logging full payloads.

Treat queued payloads as untrusted input. Validate their shape and version when a worker reads them, authorize sensitive operations inside the handler, and keep credentials in the worker’s secret store rather than in job data. Apply retention limits to completed and dead-letter records. Restrict who can inspect, replay, or delete jobs because those actions can expose data or repeat side effects.

Common pitfalls

• Putting large files or secrets directly in messages instead of storing a reference.
• Running a scheduled action on every application replica without a uniqueness guard.
• Acknowledging a job before its side effect commits.
• Retrying permanent errors or retrying without jitter.
• Raising concurrency without checking database connection limits and API rate limits.
• Replaying a dead-letter job without checking whether part of its side effect already happened.
• Keeping only queue depth and ignoring how old the oldest job has become.

Quick Recap Checklist

• Decide whether the caller can safely receive an acknowledgement before the job finishes.
• Choose a durable queue if work must survive process restarts.
• Set queue capacity, worker concurrency, and a policy for overload.
• Make handlers idempotent and define retryable versus permanent errors.
• Use durable schedule state and prevent duplicate schedule occurrences.
• Measure queue age, runtime, failures, retries, and dead-letter volume.
• Validate payloads and keep secrets out of messages.
• Stop claiming new work during shutdown and let leases recover interrupted jobs.

Interview Questions

Further Reading

• Message queues — Compare queue mechanisms and delivery behavior.
• Outbox pattern — Reliably connect database writes to published events.
• Backpressure handling — Keep producers from overwhelming workers and downstream services.
• Python asyncio queues — Review the queue API used in the in-process example.
• Celery task guide — See task retry and acknowledgement behavior in a durable worker system.

Conclusion

Background work is easier to operate when its limits are visible. Keep the queue bounded, cap worker concurrency, make retries safe, and decide how scheduled work behaves after downtime. Start with the failure cases you can explain to an on-call engineer; then choose a broker or scheduler that fits those requirements.