observability-and-instrumentation

Emit JSON log events with stable event names and machine-readable fields instead of string interpolation. Attach a correlation ID via a child logger per request, and use consistent levels: error for broken invariants, warn for handled degradation, info for business events.

Use the RED method: Rate (requests per second), Errors (failure rate), and Duration (latency histogram with p50/p95/p99 percentiles). Apply it to every endpoint and external dependency, keeping labels to bounded sets like route template and status class.

OpenTelemetry is the vendor-neutral standard for tracing and metrics with auto-instrumentation for HTTP, gRPC, and database clients. Prometheus prom-client is one common metrics backend; the RED/USE and cardinality rules apply identically either way.

Every unique label combination creates a separate time series, so labels like user IDs, raw URLs, or error messages explode storage and query costs. Keep labels to small fixed sets and put high-cardinality values in logs and traces instead.

Page only for user-facing symptoms requiring immediate action, such as error rate above 1% for 5 minutes. Use ticket severity for degradation that can wait a week, and delete any alert whose response is to ignore it.

Do not use it to diagnose an active failure (use a debugging skill), profile measured slowness (use performance optimization), or build launch-day monitoring checklists. Instrumentation is written alongside the feature, before those situations arise.