resonance-ops-observability

Start with the Four Golden Signals or RED: Rate, Errors, and Duration per endpoint. Use histograms for latency percentiles split by success and error, add structured logs carrying trace_id, and instrument OpenTelemetry tracing so slow or failed requests can be localized in the call graph.

Define an SLI as a ratio of good to valid events measured close to the user, then set an SLO target like 99.9% over 28 days based on what users actually tolerate. The error budget is 100% minus the SLO; budget remaining means ship features, budget exhausted means freeze and fix reliability.

No, alert on user-facing symptoms like error rate and latency, not causes like CPU or pod restarts. A system can run hot and still serve every request, so cause-based pages train on-call to ignore the pager; keep causes on diagnostic dashboards instead.

Metrics cost scales per unique label combination, so one unbounded label like raw user_id or full URL can explode into millions of series and take down the metrics backend. Put high-cardinality detail on traces and logs, and use route patterns instead of concrete paths on metrics.

No, never log secrets, tokens, Authorization headers, card numbers, or full request and response bodies. Structured logging makes leaks searchable, so redact at the source and instead log route, status, and latency correlated by trace_id.

Delete or downgrade alerts that never lead to action, group related alerts into one incident, and page only on SLO burn rate and Golden Signals. Every alert must name its symptom, user impact, and first response step, or it becomes a dashboard or ticket instead.