Message Queues and Asynchronous Processing
Message queues decouple the production and consumption of work — enabling components of a system to communicate without being directly connected and without requiring both to be available simultaneously. Asynchronous processing via message queues is essential for building resilient, scalable systems.
Why Message Queues
- Decoupling: The component that creates work (producer) and the component that does work (consumer) don't need to know about each other directly
- Resilience: If a consumer fails, messages remain in the queue and are processed when it recovers — no data loss
- Load levelling: Producers can burst without overwhelming consumers — the queue absorbs the burst and consumers process at a sustainable rate
- Scalability: Add more consumers to process the queue faster — simple horizontal scaling for background work
When to Use Async Processing
- Email sending — don't make users wait for SMTP
- Image processing and document generation
- Third-party API calls that don't need immediate results
- Data synchronisation and exports
- Notifications and webhooks delivery
- Any operation taking more than a few hundred milliseconds that doesn't need an immediate user response
Technologies
- AWS SQS: Managed queue service — simple, highly reliable, excellent AWS integration
- AWS SNS + SQS: Pub/sub pattern — one event fans out to multiple queues
- Redis (BullMQ, Sidekiq): Application-level job queues using Redis
- RabbitMQ: Open-source message broker with rich routing and exchange patterns
- Apache Kafka: High-throughput event streaming — appropriate for very high volume event processing