skills.rest
incidentfox
incidentfoxOFFICIAL
Explore all skills by this author →
WebsiteGitHub

kafka-streaming

Inspect Kafka brokers, topics, consumer groups, and consumer lag via Python scripts.

656|82|Updated Jan 20, 2026
Search Tags:#Streaming#Kafka#consumer groups#consumer lag#Incident Investigation#confluent-kafka#topic inspection
One-click install
npx skills add https://github.com/incidentfox/incidentfox --skill kafka-streaming
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: kafka-streaming
Source: https://github.com/incidentfox/incidentfox/tree/main/sre-agent/.claude/skills/streaming-kafka
Command: npx skills add https://github.com/incidentfox/incidentfox --skill kafka-streaming

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires confluent-kafka, and includes scripts (resource) components.

What problem does it solve?

Investigating Kafka issues like consumer lag, under-replicated partitions, or unhealthy brokers normally requires juggling multiple CLI commands and manual correlation. This Skill provides a structured, broker-first workflow that surfaces cluster health, topic details, and consumer group status as clean JSON output.

Core Features & Use Cases

  • Broker and Cluster Inspection: Retrieve cluster ID, controller, and broker inventory as the mandatory first step of any investigation.
  • Topic Analysis: List topics and describe partition layout, replication, configs, and per-partition offsets, including under-replicated partition detection.
  • Consumer Lag Monitoring: Compute per-partition lag for a consumer group and classify health as healthy, minor_lag, lagging, or severely_lagging.
  • Use Case: An on-call engineer sees delayed event processing. They run the lag workflow to identify the affected consumer group, find the partitions with the highest lag, and check whether under-replicated partitions on the topic are the root cause.

Quick Start

Check the consumer lag for my Kafka consumer group 'order-processor' and tell me if it is healthy.

Frequently Asked Questions about kafka-streaming

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check Kafka consumer lag for a consumer group?▼

Run get_consumer_lag.py with the --group flag to compute per-partition lag by comparing committed offsets against high watermarks. The output includes total lag and a health classification of healthy, minor_lag, lagging, or severely_lagging.

How to list and describe Kafka topics from Python?▼

Use list_topics.py to enumerate topics with partition counts, then describe_topic.py --topic NAME for details. The describe output includes partition leaders, replicas, ISRs, topic configs, and per-partition offset ranges.

Does this Kafka tooling support SASL and SSL authentication?▼

Yes, the shared kafka_client.py supports PLAINTEXT, SSL, SASL_SSL, and SASL_PLAINTEXT protocols via KAFKA_SECURITY_PROTOCOL. Credentials are injected transparently by a proxy layer, so scripts run without manual credential handling.

Why does my Kafka consumer group show no committed offsets?▼

The lag script returns an empty result when the group has no committed offsets, which happens for new groups or groups that never committed. Verify the group ID with list_consumer_groups.py and confirm consumers are actively committing offsets.

What Kafka lag threshold indicates a problem?▼

This tooling classifies total lag of zero as healthy, under 1,000 as minor_lag, under 100,000 as lagging, and 100,000 or more as severely_lagging. Thresholds are fixed in the script and applied to the summed lag across partitions.

Related Skills

Explore More Software Engineering Skills→
metalbear-cometalbear-co
26

mirrord-kafka

Automate Kafka queue splitting configuration in Kubernetes with CRDs and mirrord.json filters.

Official
Advanced
Greenmamba29Greenmamba29

slack-qa-investigate

Investigate Slack QA issues by correlating thread context with deployment logs and error metrics.

Community
Advanced
ivanshamaevivanshamaev
14

infra-kafka-platform-review

Review Kafka production configurations for broker, topic, consumer, and monitoring readiness.

Community
Advanced
armoin2018armoin2018
4

ailey-data-kafka

Automate Kafka workflows for real-time data pipelines across Confluent Cloud, Platform, Apache Kafka, and AWS MSK.

Community
Advanced