observability-and-instrumentation

Structured logging means emitting JSON log events with stable event names and machine-readable fields instead of interpolated strings. Attach a correlation or request ID at the system boundary and include it on every log line, and never log secrets, tokens, or full PII.

Use the RED method: Rate (requests per second), Errors (failure rate), and Duration (latency histogram). For resources like queues and pools, use USE: Utilization, Saturation, Errors. Track percentiles like p95 and p99 rather than averages.

OpenTelemetry is the vendor-neutral standard for tracing and metrics with auto-instrumentation for HTTP, gRPC, and common database clients. Prometheus prom-client is one common metrics backend; the RED/USE and cardinality rules apply identically either way.

Every unique label combination creates a separate time series, so labels like user IDs, raw URLs, or error messages explode storage and query costs. Labels must come from small fixed sets such as route templates, status classes, and provider names; unbounded values belong in logs and traces.

Alert on symptoms users feel, such as error rate above 1% or p99 latency above 2 seconds, because they fire exactly when users are hurt regardless of cause. Cause-based alerts like CPU at 85% fire when nothing is wrong and miss unpredicted failures.

It is not for diagnosing an active failure, which belongs to debugging and error recovery, nor for profiling measured slowness, which belongs to performance optimization. It covers writing instrumentation alongside features, not launch-day monitoring checklists.