streaming-data

Use the KafkaJS library to create a producer with idempotence enabled and acks set to all, then send messages with a key for partitioning. For consumers, subscribe to topics, process each message, and manually commit offsets only after successful processing.

Choose Kafka for a mature ecosystem and event replay, Redpanda for lower latency and simpler operations without ZooKeeper, and Pulsar for native multi-tenancy and geo-replication. RabbitMQ fits traditional task queues rather than event streaming.

At-least-once guarantees no message loss but may produce duplicates, requiring idempotent consumers. Exactly-once uses idempotent producers and transactions to prevent both loss and duplicates, at the cost of higher latency and complexity.

Enable binary logging in MySQL or logical replication in PostgreSQL, create a replication user, then deploy a Debezium connector configuration pointing at your database and Kafka cluster. Row-level changes are published as events to Kafka topics.

Consumer lag grows when processing is slower than message production or partitions outnumber consumers. Match partition count to consumer instances, reduce per-message processing time, and monitor lag metrics to catch bottlenecks early.

Use a dead letter queue when messages fail parsing, exceed maximum retry attempts, or hit permanently unavailable downstream services. Failed messages are routed to a separate topic with error metadata for later analysis instead of blocking the pipeline.