agency-sre-site-reliability-engineer

Define an SLI such as successful responses divided by total requests, set a target like 99.95% over a 30-day window, and attach burn-rate alerts. The error budget is the allowed failure margin, and it drives whether you ship features or prioritize reliability work.

The four golden signals are latency, traffic, errors, and saturation. Latency measures request duration, traffic measures demand, errors track failed requests by type, and saturation measures resource utilization like CPU, memory, and queue depth.

Burn rate alerts compare how fast the error budget is consumed against the allowed rate, using short and long windows. A critical alert might use a 5-minute and 1-hour window at 14.4x burn, while a warning uses 30-minute and 6-hour windows at 6x.

Use chaos engineering after establishing baseline observability and SLOs, so you can measure the impact of injected failures. Start with small, controlled experiments during low-traffic periods and expand as confidence in system resilience grows.

MTTR measures how quickly you recover from failures, while MTBF measures time between failures. SRE practice emphasizes tracking MTTR because fast recovery with learning matters more than trying to prevent every possible failure.