Message queue
Also known as: message broker, job queue, task queue, messaging queue
A message queue is software that temporarily holds messages, such as orders, emails to send, or jobs to process, passed from one part of a system to another.
The sender drops messages in the queue and moves on, while the receiver processes them at its own pace. This keeps systems responsive during traffic spikes and prevents data loss if one part fails. Examples include RabbitMQ, Amazon SQS, Azure Service Bus, and Apache Kafka. While queues add reliability, they also add complexity, such as handling duplicate or stuck messages.
Message queue, explained at three levels
At home
A message queue is like a ticket line at a deli. Customers take a number, and staff serve them in order, even when the shop gets busy.
At work
Queues stop systems from crashing under heavy load by letting work wait its turn.
At home
When you buy concert tickets online and land in a 'virtual waiting room,' a queue is managing the rush behind the scenes.
At work
Queues decouple systems, so a slow email service doesn't slow down checkout. Features include retries, dead-letter queues for failed messages, and message ordering options.
At home
If you get a duplicate confirmation email, a queue may have retried a message. In my opinion, that's usually better than getting no email at all.
At work
Design consumers to be idempotent, monitor queue depth and message age, alert on dead-letter queues, secure queue access, and decide on 'at-least-once' versus 'exactly-once' processing needs.
Real-world examples
At home
Carlos ordered limited-edition sneakers online. The site put him in a waiting queue, and his order processed smoothly instead of the site crashing.
At work
Nadia designs systems for a Toronto e-commerce company. Adding a message queue between checkout and the warehouse system kept orders flowing during Black Friday.
How it compares
| feature | Message queue | Apache Kafka |
|---|---|---|
| Main pattern | Tasks processed once | Event streams read by many |
| Message retention | Removed after processing | Kept for a set time |
| Typical use | Background jobs | Analytics, event-driven systems |
| Examples | RabbitMQ, Amazon SQS | Kafka, Confluent |
Queues hand out tasks. Kafka-style logs broadcast events.
For analysts
When mapping business processes, I'd identify where work can wait safely. Those points are natural places for queues that make systems more resilient.
History & origin
Message-oriented middleware emerged in banking and enterprise computing in the 1980s and 1990s, with IBM's MQSeries, launched in 1993, becoming a standard for connecting mainframe and business systems. The AMQP open standard began in the mid-2000s, and RabbitMQ was released in 2007. Amazon Simple Queue Service, launched in beta in 2004, was one of the first services Amazon Web Services offered. LinkedIn created Apache Kafka in 2011 for high-volume event streaming, and cloud-managed queues made the pattern easy for small teams. In my view, queues are one of the simplest ways to make software resilient.