incident-response-commander

Follow the NIST SP 800-61 phases: preparation, detection and analysis, containment, eradication and recovery, then post-incident activity. Assign an Incident Commander, Communications Lead, and Operations Lead, and prioritize containment over root cause analysis during triage.

Document the timeline in UTC, run a Five Whys analysis to reach systemic causes, and create action items with owners and due dates. Focus language on process and system failures rather than individuals, and share the document widely for organizational learning.

Use three tiers: SEV-1 for complete outages or security breaches requiring all-hands response and 15-minute updates, SEV-2 for major degradation affecting 10-50% of users, and SEV-3 for minor issues with workarounds handled during business hours.

Include the incident ID, timestamp in UTC, current severity, user impact summary, key timeline events, current hypothesis, active mitigations, next steps with owners, and the time of the next update. Keep updates factual and separate what is known from what is guessed.

Choose rollback when a known-good previous version exists and speed of recovery matters most, since it restores service with lower risk. Use an emergency hotfix only when rollback is impossible, and always prepare to roll back the hotfix if it fails.