using-message-queues_skill

Overview

This skill teaches asynchronous communication using message brokers and task queues for event-driven systems, background jobs, and service decoupling. It compares brokers (Kafka, RabbitMQ, NATS, Redis Streams), task queues (Celery, BullMQ), and orchestration tools (Temporal), and provides concrete examples and patterns for production use. The content focuses on broker selection, event schema, idempotency, DLQs, and frontend integration for job status.

How this skill works

The skill inspects use cases (throughput, latency, routing, durability) and maps them to appropriate brokers and libraries. It provides code examples for producers/consumers, task workers, and workflows, plus patterns like event naming, schema shape, DLQs, and idempotency. It also includes frontend patterns (SSE/React) for real-time job status and practical anti-patterns to avoid.

When to use it

• Long-running operations that must not block HTTP requests (reports, media processing)
• Decoupling microservices or implementing event-driven architectures and CQRS
• Guaranteed delivery or replayable event streams for payments, orders, and analytics
• Background job processing for emails, image resizing, webhooks
• Complex workflows, sagas, and human-in-the-loop orchestration
• Request-reply/RPC with low-latency microservices or IoT command/control

Best practices

• Choose broker by primary need: Kafka for streaming, RabbitMQ for complex routing, NATS for RPC, Redis Streams for simple queues, Temporal for durable workflows
• Design event names and schemas clearly (Domain.Entity.Action.Version) and include metadata for tracing and correlation
• Make consumers idempotent and use an idempotency key or dedupe store (Redis) to ensure exactly-once semantics where needed
• Implement dead letter queues and monitoring to surface persistent failures for manual inspection
• Avoid synchronous heavy work in HTTP handlers: enqueue tasks and expose status endpoints or SSE for client updates
• Use retries with exponential backoff and classify recoverable vs unrecoverable errors to route failures appropriately

Example use cases

• High-throughput analytics pipeline with Kafka for long-term retention and event sourcing
• Background image processing using Celery + Redis or BullMQ for TypeScript systems
• Order processing saga implemented with Temporal to coordinate inventory, payment, and compensating actions
• Microservice RPC and sub-ms commands using NATS request-reply
• Notification or simple job queue on existing Redis infrastructure via Redis Streams