Skip to content
Velloreth
Programming 5 min read

Background Jobs: Queues, Workers and Retries

Move slow work out of the request. Learn the queue, the worker, the retry policy, idempotency and how to monitor a background job.

By

Why a background job

A request should be fast. Work that takes seconds, that calls a third party, or that can fail and be retried belongs in a background job.

The pattern:

Request  ->  enqueue a job  ->  respond immediately
Worker   ->  dequeue  ->  do the work  ->  ack

The response is fast, and the work happens out of band.

The trade-off: the work is no longer synchronous, so the user does not see the result immediately. A job that the user needs in the response should stay in the request.

The queue

A queue is a durable list of jobs. It survives a restart, which is what makes it different from an in-memory list.

The common implementations:

SystemNotes
Redis (BullMQ, Sidekiq)fast, simple, in-memory with persistence
RabbitMQa full message broker with routing
SQSmanaged, at-least-once delivery
Postgres (pg-boss)a queue in a database you already have
Kafkaa log, for a stream rather than a task

The delivery guarantee matters. Most queues are at-least-once, which means a job can be delivered more than once. This is why a job must be idempotent.

A queue in Postgres is a good default for a small application, because it needs no new infrastructure and it participates in the same transaction as the data.

The worker

A worker is a process that dequeues a job and runs it.

const worker = new Worker('email', async (job) => {
  await sendEmail(job.data.to, job.data.subject);
}, { connection });

The worker acknowledges the job when it succeeds. If the worker crashes before the ack, the job is redelivered.

A concurrency setting controls how many jobs a worker runs at once. Too high and the worker exhausts its connections; too low and the queue grows.

A timeout is essential. A job that hangs holds a worker slot forever. Set a timeout and let the job fail and retry.

Idempotency

A job must be safe to run twice.

Why: the queue can deliver a job more than once, and a retry after a timeout can run a job that actually succeeded.

The patterns:

A unique key. Record the job’s key in a table with a unique constraint, and skip if it exists.

INSERT INTO processed_jobs (job_key) VALUES ($1)
ON CONFLICT DO NOTHING;

A state check. Read the current state and act only if the transition is valid.

if (order.status === 'pending') {
  await charge(order);
  order.status = 'paid';
}

An idempotency key on the external call. Many payment APIs accept a key and return the original result for a repeat.

A natural idempotency. Setting a value is idempotent; incrementing it is not.

Retries

A retry needs a policy.

{
  attempts: 5,
  backoff: { type: 'exponential', delay: 1000 },
}

Exponential backoff grows the delay: 1s, 2s, 4s, 8s, 16s.

A jitter randomises the delay so the retries do not align.

const delay = Math.min(cap, base * 2 ** attempt) * (0.5 + Math.random() / 2);

A maximum attempt count is required. A job that retries forever is a job that never fails and never succeeds.

Retry only a transient failure. A validation error will fail every time. Classify the error and retry only the ones that can succeed.

if (err.statusCode >= 500 || err.code === 'ETIMEDOUT') {
  throw err; // retry
}
// a 400 is permanent: do not retry

The dead-letter queue

A job that fails every attempt goes to a dead-letter queue.

Why: it keeps a failing job from blocking the queue, and it preserves the job for inspection.

A dead-letter queue needs an alert. A queue that nobody monitors is a silent failure.

The recovery: inspect the job, fix the cause, and replay it.

Scheduling

A delayed job runs after a delay.

await queue.add('reminder', data, { delay: 24 * 60 * 60 * 1000 });

A repeatable job runs on a schedule.

await queue.add('report', data, { repeat: { cron: '0 6 * * *' } });

A cron job that enqueues a queue job is a common combination: the schedule triggers the work, and the queue handles the retry and the concurrency.

Monitoring

The metrics that matter:

MetricWhy
Queue deptha growing queue means the workers cannot keep up
Age of the oldest jobthe real latency the user experiences
Success ratea drop is the first sign of a problem
Processing timea slow job reduces the throughput
Retry ratea high rate means a dependency is flaky
Dead-letter counta non-zero count needs attention

Alert on the queue depth and the age. A queue that grows overnight is a queue that will be a problem in the morning.

Log the job id and the attempt. A log without the job id cannot be correlated with the failure.

Common mistakes

A non-idempotent job. A retry charges the card twice.

No timeout. A stuck job holds a worker forever.

An infinite retry. A job that never fails and never succeeds.

Retrying a permanent error. A validation error fails every attempt and fills the dead-letter queue.

No dead-letter queue. A failing job blocks the queue or is lost.

A dead-letter queue with no alert. A silent failure.

A job that depends on the request’s in-memory state. The worker is a different process, so the state is not there. Pass the data in the job.

A large payload in the job. Pass an id and read the data in the worker.

No monitoring. The queue grows and nobody notices until the users complain.

A quotable summary

A background job moves slow work out of the request so the response is fast, using a durable queue and a worker that acknowledges the job when it succeeds. A job must be idempotent, because the queue can deliver it more than once, and a retry needs an exponential backoff with a jitter and a maximum attempt count. A job that fails every attempt goes to a dead-letter queue, which needs an alert, and the queue depth and the age of the oldest job are the metrics to watch.

For the queue patterns, see message-queues-explained and cron-vs-job-queues. For the broader context, idempotency-keys-explained and monitoring-vs-observability.

Frequently asked questions

When should work go into a background job?
When it is slow, when it can fail and be retried, or when it does not need to finish before the response. Sending an email, generating a report, resizing an image and calling a third-party API are the common cases. Work that the user needs in the response should stay in the request.
What is idempotency in a job?
It means running the job twice has the same effect as running it once. It matters because a queue can deliver a job more than once, and a retry after a timeout can run a job that actually succeeded. A job that charges a card must check whether the charge already happened.
What is a dead-letter queue?
It is a queue that holds the jobs that failed every retry attempt. It keeps a failing job from blocking the queue and preserves it for inspection. A dead-letter queue that nobody monitors is a silent failure, so it needs an alert.
What is exponential backoff?
It is a retry delay that grows with each attempt, such as 1 second, then 2, then 4, then 8. It gives a failing dependency time to recover and it reduces the load on it. A jitter is usually added so the retries do not align.
How do I monitor a background job?
Track the queue depth, the age of the oldest job, the success and failure rate, and the processing time. Alert on the queue depth and the age, because a growing queue is the first sign that the workers cannot keep up.
What is the difference between a queue and a cron job?
A cron job runs on a schedule, and a queue job runs when it is enqueued. A cron job is for a recurring task such as a nightly report, and a queue job is for a task triggered by an event. A cron job that enqueues a queue job is a common combination.