incident-response-procedures

Define severity levels (P0-P3) with response times, write triage steps for classifying incidents, and create diagnostic runbooks with specific commands for common failures. Include escalation contacts, communication templates, and rollback procedures validated before deployment.

Use four levels: P0 for complete outages requiring immediate response, P1 for major degradation within 30 minutes, P2 for partial degradation within 2 hours, and P3 for minor trends handled next business day. Each level needs clear examples and actions.

The Skill requires the optimization report, target architecture design document, monitoring setup with alert definitions, deployment artifacts like rollback procedures, and team contacts including on-call rotation and escalation paths.

Execute immediate rollback for P0 incidents like complete outages, data corruption, or security breaches. Consider rollback for P1 incidents unresolved after one hour. Avoid rollback for P2/P3 issues with known workarounds or fixes in progress.

Include incident summary with severity and duration, a timeline of events, root cause analysis using techniques like 5 Whys, impact assessment, resolution steps, and preventive actions. Complete the report within 24 hours and update runbooks with learnings.