managing-incidents

Classify incidents using impact and urgency criteria: SEV0 for complete outages, SEV1 for major degradation, SEV2 for minor issues, and SEV3 for cosmetic problems. Run the included classify-severity.py script for an interactive questionnaire that recommends a severity level.

Schedule the post-mortem within 48 hours, assign a neutral facilitator, walk through the timeline, and use 5 Whys analysis to find systemic root causes. Focus on process failures rather than individuals, and define specific action items with owners and due dates.

PagerDuty suits enterprises with complex escalation policies, Opsgenie fits Atlassian ecosystem users, and incident.io works well for Slack-native teams. Choose based on team size, budget, and existing tooling integrations.

One-week rotations provide the best balance between context retention and burnout prevention. Use a primary plus secondary model where the secondary is paged if the primary does not acknowledge within 5 minutes, and hold a 30-minute handoff call.

Escalate when there is no progress after 30 minutes, when severity increases such as SEV2 becoming SEV1, or when specialized expertise like database or security knowledge is needed. Avoid escalating when mitigation is actively progressing.