operating-production-services

Define an SLI measuring good events over total events, such as successful HTTP requests over all requests, then set a target like 99.9%. Your error budget is 1 minus the target, giving 43 minutes of allowed downtime per month at 99.9%.

Use a template with an executive summary, UTC timeline, 5 Whys root cause analysis, impact assessment, and action items with owners and due dates. Focus on what conditions allowed the failure rather than who caused it.

Multi-window burn rate alerting combines short and long measurement windows to detect error budget consumption quickly without noise. A fast-burn alert fires at 14.4x burn rate over 1 hour and 5 minutes, while slow-burn uses 6x over 6 hours.

Write postmortems for SEV1/SEV2 incidents, customer-facing outages over 15 minutes, data loss or security events, and near-misses with novel failure modes. Minor incidents can use a quick template with just timeline, root cause, and fix.

A 100% SLO leaves zero error budget, meaning any failure freezes all feature work. Each additional nine costs exponentially more, so 99.9% or 99.95% targets balance reliability with development velocity.