observability-and-instrumentation

Structured logging means emitting JSON log events with stable event names and machine-readable fields instead of interpolated strings. Attach a correlation ID at the request boundary, use consistent levels (error, warn, info, debug), and never log secrets or full PII.

RED metrics are Rate, Errors, and Duration, applied to every request-driven endpoint and external dependency. For resources like queues and pools, use USE metrics instead: Utilization, Saturation, and Errors. Track latency as histogram percentiles, not averages.

OpenTelemetry is the vendor-neutral standard for tracing and metrics, with auto-instrumentation for HTTP, gRPC, and common database clients. Prometheus client libraries like prom-client are one common metrics backend; the RED/USE and cardinality rules apply identically either way.

High-cardinality labels like user IDs, raw URLs, or error messages create a separate time series per unique value, which can overwhelm the metrics backend. Labels must come from small fixed sets such as route templates and status classes; unbounded values belong in logs and traces.

Do not use it to diagnose an active failure; use debugging skills for that, since observability is what makes future diagnosis fast. It also does not cover performance profiling of measured slowness or launch-day monitoring checklists, which belong to separate skills.