Background Jobs: Queues, Workers and Retries
Move slow work out of the request. Learn the queue, the worker, the retry policy, idempotency and how to monitor a background job.
By Velloreth
Why a background job
A request should be fast. Work that takes seconds, that calls a third party, or that can fail and be retried belongs in a background job.
The pattern:
Request -> enqueue a job -> respond immediately
Worker -> dequeue -> do the work -> ack
The response is fast, and the work happens out of band.
The trade-off: the work is no longer synchronous, so the user does not see the result immediately. A job that the user needs in the response should stay in the request.
The queue
A queue is a durable list of jobs. It survives a restart, which is what makes it different from an in-memory list.
The common implementations:
| System | Notes |
|---|---|
| Redis (BullMQ, Sidekiq) | fast, simple, in-memory with persistence |
| RabbitMQ | a full message broker with routing |
| SQS | managed, at-least-once delivery |
| Postgres (pg-boss) | a queue in a database you already have |
| Kafka | a log, for a stream rather than a task |
The delivery guarantee matters. Most queues are at-least-once, which means a job can be delivered more than once. This is why a job must be idempotent.
A queue in Postgres is a good default for a small application, because it needs no new infrastructure and it participates in the same transaction as the data.
The worker
A worker is a process that dequeues a job and runs it.
const worker = new Worker('email', async (job) => {
await sendEmail(job.data.to, job.data.subject);
}, { connection });
The worker acknowledges the job when it succeeds. If the worker crashes before the ack, the job is redelivered.
A concurrency setting controls how many jobs a worker runs at once. Too high and the worker exhausts its connections; too low and the queue grows.
A timeout is essential. A job that hangs holds a worker slot forever. Set a timeout and let the job fail and retry.
Idempotency
A job must be safe to run twice.
Why: the queue can deliver a job more than once, and a retry after a timeout can run a job that actually succeeded.
The patterns:
A unique key. Record the job’s key in a table with a unique constraint, and skip if it exists.
INSERT INTO processed_jobs (job_key) VALUES ($1)
ON CONFLICT DO NOTHING;
A state check. Read the current state and act only if the transition is valid.
if (order.status === 'pending') {
await charge(order);
order.status = 'paid';
}
An idempotency key on the external call. Many payment APIs accept a key and return the original result for a repeat.
A natural idempotency. Setting a value is idempotent; incrementing it is not.
Retries
A retry needs a policy.
{
attempts: 5,
backoff: { type: 'exponential', delay: 1000 },
}
Exponential backoff grows the delay: 1s, 2s, 4s, 8s, 16s.
A jitter randomises the delay so the retries do not align.
const delay = Math.min(cap, base * 2 ** attempt) * (0.5 + Math.random() / 2);
A maximum attempt count is required. A job that retries forever is a job that never fails and never succeeds.
Retry only a transient failure. A validation error will fail every time. Classify the error and retry only the ones that can succeed.
if (err.statusCode >= 500 || err.code === 'ETIMEDOUT') {
throw err; // retry
}
// a 400 is permanent: do not retry
The dead-letter queue
A job that fails every attempt goes to a dead-letter queue.
Why: it keeps a failing job from blocking the queue, and it preserves the job for inspection.
A dead-letter queue needs an alert. A queue that nobody monitors is a silent failure.
The recovery: inspect the job, fix the cause, and replay it.
Scheduling
A delayed job runs after a delay.
await queue.add('reminder', data, { delay: 24 * 60 * 60 * 1000 });
A repeatable job runs on a schedule.
await queue.add('report', data, { repeat: { cron: '0 6 * * *' } });
A cron job that enqueues a queue job is a common combination: the schedule triggers the work, and the queue handles the retry and the concurrency.
Monitoring
The metrics that matter:
| Metric | Why |
|---|---|
| Queue depth | a growing queue means the workers cannot keep up |
| Age of the oldest job | the real latency the user experiences |
| Success rate | a drop is the first sign of a problem |
| Processing time | a slow job reduces the throughput |
| Retry rate | a high rate means a dependency is flaky |
| Dead-letter count | a non-zero count needs attention |
Alert on the queue depth and the age. A queue that grows overnight is a queue that will be a problem in the morning.
Log the job id and the attempt. A log without the job id cannot be correlated with the failure.
Common mistakes
A non-idempotent job. A retry charges the card twice.
No timeout. A stuck job holds a worker forever.
An infinite retry. A job that never fails and never succeeds.
Retrying a permanent error. A validation error fails every attempt and fills the dead-letter queue.
No dead-letter queue. A failing job blocks the queue or is lost.
A dead-letter queue with no alert. A silent failure.
A job that depends on the request’s in-memory state. The worker is a different process, so the state is not there. Pass the data in the job.
A large payload in the job. Pass an id and read the data in the worker.
No monitoring. The queue grows and nobody notices until the users complain.
A quotable summary
A background job moves slow work out of the request so the response is fast, using a durable queue and a worker that acknowledges the job when it succeeds. A job must be idempotent, because the queue can deliver it more than once, and a retry needs an exponential backoff with a jitter and a maximum attempt count. A job that fails every attempt goes to a dead-letter queue, which needs an alert, and the queue depth and the age of the oldest job are the metrics to watch.
Related reading
For the queue patterns, see message-queues-explained and cron-vs-job-queues. For the broader context, idempotency-keys-explained and monitoring-vs-observability.