production-debugging

Diagnose live system failures using logs, metrics, and traces without a local repro.

1|Updated Jul 3, 2026
One-click install
npx skills add https://github.com/Nandansai08/skillz --skill production-debugging-nandansai08
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: production-debugging
Source: https://github.com/Nandansai08/skillz/tree/main/skills/sre-incident-response/production-debugging
Command: npx skills add https://github.com/Nandansai08/skillz --skill production-debugging-nandansai08

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Production systems fail in ways that cannot be reproduced locally, and rushed fixes like restarts or fleet-wide debug logging destroy evidence or cause new outages. This Skill provides a disciplined workflow for root-causing live misbehavior — error spikes, latency, resource leaks — without making production worse. ## Core Features & Use Cases - Change-anchored investigation: Line up symptom onset against deploys, config changes, cron jobs, and data thresholds to find what changed at the failure's start time. - Failure-shape characterization: Slice errored requests by endpoint, region, instance, and customer so the partition itself names the suspect before any hypothesis is formed. - Metrics-logs-traces funnel: Follow one exemplar request through all three telemetry layers, read latency as distributions (p50 vs p99), and test one falsifiable hypothesis at a time. - Use Case: p99 latency on /search jumps 8x overnight with p50 flat. The workflow traces it to a nightly reindex leaving one hot Elasticsearch shard, confirms it via segment counts without touching anything, and ships a post-reindex alert as follow-up. ## Quick Start Use the production-debugging skill to diagnose why our API's p99 latency spiked after last night's deploy using our logs, metrics, and traces.

Frequently Asked Questions about production-debugging

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I debug a production issue I can't reproduce locally?▼

Anchor on what changed at the symptom's start time, then characterize the failure's shape by slicing errored requests across endpoint, region, and instance dimensions. Follow one exemplar request through metrics, logs, and traces to locate the failing hop before forming hypotheses.

How to find the root cause of a latency spike in production?▼

Read latency as a distribution, not an average: flat p50 with exploding p99 indicates a subset problem like lock contention, GC pauses, or one slow shard. Trace an exemplar slow request to identify which hop consumed the time, then test one falsifiable hypothesis.

Should I restart a misbehaving production service before debugging?▼

No. Restarting destroys heap state, connection tables, and thread stacks — the core evidence. Capture a heap or thread dump from one instance first, then restart if mitigation requires it.

When should I use incident triage instead of production debugging?▼

During an active outage with users down, run incident-triage first to mitigate impact. Production debugging applies after mitigation, when root-cause hunting can proceed without time pressure.

Why is turning on debug logging across all servers risky?▼

Fleet-wide DEBUG logging can melt the log pipeline and fill disks, turning observation into a second outage. Increase log level on one instance only, time-boxed and announced, escalating to profilers or tcpdump only as a last resort.