alerting-design

Audit every paging alert by asking what the responder did the last five times it fired; if the answer is just acknowledged it, demote it to a ticket or dashboard. Then review monthly using pages per on-call week, actionable rate, and postmortem detection gaps.

Use multi-window burn-rate alerting instead of static thresholds: page when the error budget burns fast (for example 14.4x over 1 hour and 5 minutes) and ticket on slow burn (3x over 24 hours). This ties every page to the risk of missing the SLO.

Page on symptoms users experience, such as error rate and latency on critical journeys, and route causes like CPU, disk, or restarts to tickets or dashboards. Cause alerts page for non-problems and miss failures whose cause you did not predict.

Fire alerts synthetically by killing a canary instance, injecting errors in staging, or unit-testing the rules. Also add a dead-man's switch that fires when the metrics pipeline stops reporting, since silence and health look identical without it.

Page only when all three hold: it is a user-facing symptom, it needs action within minutes, and a human can act. Issues needing action within days, like a disk at 70% or a cert expiring in two weeks, belong in the ticket lane.