Message Queues Explained: Decoupling Services
A queue decouples a producer from a consumer. Learn the delivery guarantee, the dead letter, the ordering, and the backpressure.
By Velloreth
The purpose
Decouple a producer from a consumer in time.
Absorb a burst of traffic.
Scale the consumer independently.
Retry a failure without losing the work.
The pattern
Producer -> [ queue ] -> Consumer
The producer does not wait for the consumer.
The consumer processes at its own rate.
The delivery guarantee
| Guarantee | Loss | Duplicate |
|---|---|---|
| At-most-once | possible | no |
| At-least-once | no | possible |
| Exactly-once | no | no |
At-least-once is the common default.
A consumer must be idempotent, because a duplicate is possible.
Exactly-once usually means at-least-once plus an idempotent consumer.
The acknowledgement
The consumer acknowledges a message after it is processed.
1. The consumer receives the message
2. The consumer processes it
3. The consumer acknowledges it
4. The queue removes it
A message that is not acknowledged is redelivered after a visibility timeout.
A long process needs a longer timeout, or the message is redelivered while it is still running.
The dead letter queue
A message that failed repeatedly is moved aside.
max_attempts: 3
dead_letter_queue: orders-dlq
It does not block the main queue.
It can be inspected and replayed.
It is the difference between a lost message and a recoverable one.
An alert on the dead letter queue is essential, because a message that sits there is a silent failure.
The ordering
Ordering is per partition or per key, not global.
A message with the same key goes to the same partition.
key=user-1 -> partition 0
key=user-2 -> partition 1
The order is preserved for a key, and not across the keys.
A global order requires a single consumer, which limits the throughput.
The backpressure
A signal that the consumer cannot keep up.
A bounded queue applies it by rejecting or blocking a new message.
An unbounded queue hides the problem until the memory or the disk is exhausted.
The metrics to watch:
- The queue depth
- The consumer lag
- The processing time
- The error rate
A growing queue depth is the signal that the consumer needs to scale.
The patterns
| Pattern | Description |
|---|---|
| A work queue | one consumer per message |
| Publish-subscribe | every subscriber gets a copy |
| A fan-out | one message to several queues |
| A request-reply | a reply queue |
| A priority queue | a high-priority message first |
| A delay queue | a message after a delay |
A work queue is the common case for a background job.
A fan-out is used to notify several systems.
The tools
| Tool | Notes |
|---|---|
| RabbitMQ | a broker with routing |
| Kafka | a distributed log |
| SQS | a managed queue |
| Redis | a simple list or stream |
| NATS | a lightweight broker |
Kafka is a log, so a message is retained and several consumer groups read it.
SQS is a managed queue with a visibility timeout.
The idempotency
A consumer must be idempotent, because a duplicate is possible.
The approaches:
- A unique constraint on a business key
- An idempotency key stored with the result
- A check that the work is already done
A non-idempotent consumer charges twice, sends twice, or creates a duplicate.
Common mistakes
A non-idempotent consumer. A duplicate.
No dead letter queue. A lost message.
No alert on the dead letter queue. A silent failure.
A visibility timeout that is too short. A redelivery.
An unbounded queue. A memory exhaustion.
No ordering key. A reordered message.
A global order assumed. A wrong result.
No consumer lag metric. A hidden backlog.
A message with a large payload. A slow queue.
No schema on the message. A breaking change.
A quotable summary
A message queue decouples a producer from a consumer in time, absorbs a burst, and lets the consumer scale independently. At-least-once is the common guarantee, so a consumer must be idempotent, and a dead letter queue holds a message that failed repeatedly so it is recoverable rather than lost. Ordering is per partition or per key rather than global, and a bounded queue applies backpressure so an overload is visible.
Related reading
For the related topics, see pub-sub-pattern-explained and background-jobs-guide. For the broader context, idempotency-keys-explained and cron-vs-job-queues.