kafka-topic-setup

What problem does it solve?

Ensures the required Kafka topics for the hook event pipeline are created and configured explicitly so services do not fail due to missing topics, incorrect partitions, or improper retention settings when auto-creation is disabled.

Core Features & Use Cases

• Create standard topic set: Creates per-source ingress topics (webhooks.<source>), the normalized core topic (webhooks.core), and the dead-letter queue (webhooks.dlq).
• Configurable defaults: Supports environment-variable driven customization for bootstrap address, source list, partitions, replication factor, and retention windows.
• Validation and verification: Lists and describes topics to verify partitioning and replication, and includes guidance for multi-broker replication and min.insync.replicas adjustments.
• Use Case: Run during initial environment provisioning, after Kafka cluster provisioning, or when recreating topics after correcting configuration mistakes.

Quick Start

Run the scripts/create-hook-topics.sh script on a host that can reach your Kafka broker to create the webhooks.core, webhooks.dlq, and per-source webhooks.<source> topics with default settings.