kafka-data-engineer

What problem does it solve?

This Skill helps design reliable, scalable event-driven pipelines using Kafka, including topics, schemas, and processing logic for asynchronous tasks, while promoting consistent architecture across services.

Core Features & Use Cases

• Topic & Schema Design: Define topics, schemas, and versioning to ensure backward compatibility in evolving systems.
• Reliable Processing: Establish idempotent handlers and at-least-once delivery with clear retry and dead-letter strategies.
• Observability: Monitor consumer lag, throughput, and KPI health to keep pipelines healthy.
• Use Case: Model a 'task-events' topic that carries TaskEvent payloads to drive real-time dashboards and notifications.

Quick Start

Outline the 'task-events' topic, its key/schema, and a basic dead-letter workflow for failed messages.