Skip to content
Velloreth
Technology 4 min read

Message Queues Explained: Decoupling Services

A queue decouples a producer from a consumer. Learn the delivery guarantee, the dead letter, the ordering, and the backpressure.

By

The purpose

Decouple a producer from a consumer in time.

Absorb a burst of traffic.

Scale the consumer independently.

Retry a failure without losing the work.

The pattern

Producer -> [ queue ] -> Consumer

The producer does not wait for the consumer.

The consumer processes at its own rate.

The delivery guarantee

GuaranteeLossDuplicate
At-most-oncepossibleno
At-least-oncenopossible
Exactly-oncenono

At-least-once is the common default.

A consumer must be idempotent, because a duplicate is possible.

Exactly-once usually means at-least-once plus an idempotent consumer.

The acknowledgement

The consumer acknowledges a message after it is processed.

1. The consumer receives the message
2. The consumer processes it
3. The consumer acknowledges it
4. The queue removes it

A message that is not acknowledged is redelivered after a visibility timeout.

A long process needs a longer timeout, or the message is redelivered while it is still running.

The dead letter queue

A message that failed repeatedly is moved aside.

max_attempts: 3
dead_letter_queue: orders-dlq

It does not block the main queue.

It can be inspected and replayed.

It is the difference between a lost message and a recoverable one.

An alert on the dead letter queue is essential, because a message that sits there is a silent failure.

The ordering

Ordering is per partition or per key, not global.

A message with the same key goes to the same partition.

key=user-1 -> partition 0
key=user-2 -> partition 1

The order is preserved for a key, and not across the keys.

A global order requires a single consumer, which limits the throughput.

The backpressure

A signal that the consumer cannot keep up.

A bounded queue applies it by rejecting or blocking a new message.

An unbounded queue hides the problem until the memory or the disk is exhausted.

The metrics to watch:

  • The queue depth
  • The consumer lag
  • The processing time
  • The error rate

A growing queue depth is the signal that the consumer needs to scale.

The patterns

PatternDescription
A work queueone consumer per message
Publish-subscribeevery subscriber gets a copy
A fan-outone message to several queues
A request-replya reply queue
A priority queuea high-priority message first
A delay queuea message after a delay

A work queue is the common case for a background job.

A fan-out is used to notify several systems.

The tools

ToolNotes
RabbitMQa broker with routing
Kafkaa distributed log
SQSa managed queue
Redisa simple list or stream
NATSa lightweight broker

Kafka is a log, so a message is retained and several consumer groups read it.

SQS is a managed queue with a visibility timeout.

The idempotency

A consumer must be idempotent, because a duplicate is possible.

The approaches:

  • A unique constraint on a business key
  • An idempotency key stored with the result
  • A check that the work is already done

A non-idempotent consumer charges twice, sends twice, or creates a duplicate.

Common mistakes

A non-idempotent consumer. A duplicate.

No dead letter queue. A lost message.

No alert on the dead letter queue. A silent failure.

A visibility timeout that is too short. A redelivery.

An unbounded queue. A memory exhaustion.

No ordering key. A reordered message.

A global order assumed. A wrong result.

No consumer lag metric. A hidden backlog.

A message with a large payload. A slow queue.

No schema on the message. A breaking change.

A quotable summary

A message queue decouples a producer from a consumer in time, absorbs a burst, and lets the consumer scale independently. At-least-once is the common guarantee, so a consumer must be idempotent, and a dead letter queue holds a message that failed repeatedly so it is recoverable rather than lost. Ordering is per partition or per key rather than global, and a bounded queue applies backpressure so an overload is visible.

For the related topics, see pub-sub-pattern-explained and background-jobs-guide. For the broader context, idempotency-keys-explained and cron-vs-job-queues.

Frequently asked questions

What is a message queue?
A buffer that holds a message from a producer until a consumer is ready to process it. It decouples the two in time, so the producer does not wait and the consumer can be scaled independently. It also absorbs a burst of traffic.
What is the difference between at-most-once, at-least-once and exactly-once?
At-most-once may lose a message and never duplicates. At-least-once never loses a message and may duplicate, which is the common default. Exactly-once is the hardest and usually means at-least-once plus an idempotent consumer, because a true exactly-once delivery is not achievable in general.
What is a dead letter queue?
A queue that holds a message that failed repeatedly, so it does not block the main queue and can be inspected. A message is moved there after a configured number of attempts. It is the difference between a lost message and a recoverable one.
What is the difference between a queue and a topic?
A queue delivers each message to one consumer, which is the work-queue pattern. A topic delivers each message to every subscriber, which is the publish-subscribe pattern. A system such as Kafka is a log that several consumer groups read independently.
What is backpressure?
A signal that the consumer cannot keep up, so the producer must slow down or the queue must grow. A bounded queue applies it by rejecting or blocking a new message. An unbounded queue hides the problem until the memory or the disk is exhausted.
How do I handle ordering?
Ordering is per partition or per key, not global. A message with the same key goes to the same partition, so the order is preserved for that key. A global order requires a single consumer, which limits the throughput.