observability

Use structlog with a TimeStamper and JSONRenderer processor configured once at startup. Bind context like service name and request_id, then emit logs as JSON with mandatory fields including timestamp, level, trace_id, and message.

Use a Histogram for latency metrics, not a Summary. Histograms can be aggregated across instances in PromQL using histogram_quantile, while Summaries calculate quantiles client-side and cannot be aggregated across replicas.

Each unique label combination creates a new time series, so labels like user_id or raw URL paths can generate millions of series and exhaust memory. Use only low-cardinality labels such as HTTP method, normalized endpoint, and status code.

Use the W3C traceparent header for HTTP and gRPC calls, and generate the request_id once at the edge. Pass it downstream via X-Request-ID headers and embed correlation_id in message payloads for async queues.

Burn rate measures how fast you consume your error budget relative to the sustainable pace. Alert at 14x burn over 1 hour or 6x over 6 hours as critical, and at 3x over 1 day or 1x over 3 days as warnings.

Alert on symptoms like error rate and latency that directly affect users, not causes like CPU usage which can spike without user impact. Use resource metrics only for capacity forecasting, and always add a for: duration to avoid flapping.