Celery

Celery runs background tasks. An application enqueues work through a broker — usually Redis or RabbitMQ — and separate worker processes consume and execute it. The API returns immediately with an identifier rather than the result.

The reason to reach for it is that some work simply cannot happen inside a request. Document ingestion, metadata extraction, report generation, sending mail, anything calling a slow external service: doing these inline means a request that takes minutes, a client that times out, and a worker process blocked for the duration. With a handful of workers, a few such requests exhaust the pool and the whole application stops responding — including for users doing nothing expensive.

The change in system shape

Adopting Celery converts a synchronous system into a distributed one, and that is a larger step than the API suggests.

The result contract changes. The client no longer receives an answer; it receives a task ID and must poll, subscribe, or be notified. Every interface touching that work has to handle a pending state, and designing that well — what the user sees while waiting, what happens if they navigate away — is more work than moving the function.

Failure becomes explicit. An inline function either returns or raises. A task can fail after partial completion, be retried, be retried after already succeeding once because the acknowledgement was lost, or vanish because a worker died. Retry semantics have to be decided rather than inherited, and tasks need to be idempotent, because at-least-once delivery means running twice is a normal occurrence rather than an error.

Observability stops being free. A failing request appears in the logs of the process handling it. A failing task fails somewhere else, possibly hours later, and without deliberate instrumentation nobody finds out. Monitoring the queue — depth, age of the oldest item, failure rate — is not optional, and a queue growing faster than it drains is the failure that matters most and is least visible.

Where the boundary goes

The judgment is what stays synchronous. Anything the user must see the result of before continuing should stay inline even if slow; anything that can complete later should not block. Getting this wrong in the direction of too much asynchrony produces an application that feels unpredictable — actions appear to succeed and their effects arrive at unrelated times.

For a smaller application, a simpler queue or a scheduled job is often enough. Celery is worth its weight when there is real volume and varied task types, and it is a lot of machinery for sending occasional email.

See also: Redis as broker, FastAPI, and Docker Compose for running workers alongside the application.