Message Queue

Explanation

In synchronous communication (HTTP), the caller waits for a response. If the called service is slow or down, the caller is blocked. Message queues make communication asynchronous: the producer writes to the queue and continues immediately; the consumer processes at its own pace. Key benefits: decoupling (producer doesn't know or care about the consumer), resilience (if consumer is down, messages accumulate and are processed on recovery — no messages lost), load leveling (traffic spikes write many messages; the consumer processes at sustainable rate), and retry handling (failed processing is retried automatically). Common patterns: work queue (one message processed by one worker — task distribution), pub/sub (one message delivered to all subscribers — event broadcasting, fan-out), competing consumers (multiple consumers pull from one queue — parallel processing, horizontal scale). Popular systems: RabbitMQ (AMQP protocol, flexible routing), Apache Kafka (log-based, high-throughput, messages retained for replay — de facto standard for event streaming), AWS SQS (managed, simple, scalable), Redis Streams (lightweight, built into Redis). Kafka's replay ability makes it especially powerful: consumers can reprocess the entire history of events.

Code Example

// BullMQ: Redis-backed job queue (Node.js)
const { Queue, Worker } = require('bullmq');
const connection = { host: 'localhost', port: 6379 };

const emailQueue = new Queue('emails', { connection });

// Producer: fire-and-forget, returns immediately
async function onUserSignup(user) {
  await emailQueue.add('welcome-email', {
    to: user.email,
    name: user.name,
  }, {
    attempts: 3,
    backoff: { type: 'exponential', delay: 1000 },
    removeOnComplete: true,
  });
  // HTTP response sent now — email processed asynchronously
}

// Consumer: runs in separate process
const emailWorker = new Worker('emails', async (job) => {
  if (job.name === 'welcome-email') {
    await sendEmail(job.data.to, `Welcome, ${job.data.name}!`);
  }
}, { connection });

emailWorker.on('failed', (job, err) => {
  console.error(`Job ${job.id} failed: ${err.message}`);
  // After 3 attempts, job moves to failed queue for inspection
});

Why It Matters for Engineers

Message queues appear in system design interviews for nearly every non-trivial system. 'Design a notification system,' 'design a task scheduler,' 'design a data pipeline' — all involve queues. Understanding trade-offs between Kafka, RabbitMQ, and SQS demonstrates production architecture knowledge. In real systems, queues solve the problem of processing work after an HTTP response (sending emails, generating PDFs, resizing images) without making users wait. This pattern is everywhere in production backends.