using-message-queues

Choose Kafka for event streaming and log aggregation at 500K+ msg/s with replay capability. Choose RabbitMQ for complex routing with exchanges and dead letter support. Choose Redis Streams for simple queues when you already run Redis and need moderate throughput.

Use Celery with Redis as the broker: define tasks with the @app.task decorator, enqueue them from your API endpoint with .delay(), and return a task ID immediately. Clients poll a status endpoint or receive updates via Server-Sent Events.

Celery handles simple background tasks and periodic jobs with retries, while Temporal provides durable workflow execution that survives restarts, supports saga patterns with compensation, and handles human-in-the-loop processes. Use Temporal for multi-step distributed transactions.

Implement idempotency by storing processed message IDs in Redis with a TTL and checking before processing. For Kafka, enable idempotent producers and manual offset commits after successful processing to achieve exactly-once semantics.

Avoid Kafka for request-reply patterns since it is asynchronous and lacks response correlation; use NATS or gRPC instead. It is also overkill for simple background job queues where Celery or BullMQ with Redis is simpler to operate.

Route failed messages to a dead letter queue after exhausting retries, then commit the offset so the message is not redelivered. Monitor DLQ depth with alerts and provide a runbook for manual inspection and replay of failed messages.