observability-and-instrumentation

Structured logging emits JSON objects with a stable event name and machine-readable fields instead of interpolated strings. Create a child logger per request carrying a correlation ID, and attach fields like error codes and attempt counts so logs are queryable.

Use the RED method: rate, errors, and duration for every endpoint and external dependency. Track latency as a histogram so p95 and p99 are queryable, and keep label sets bounded to values like route templates and status classes.

Initialize the OpenTelemetry NodeSDK with auto-instrumentations before importing anything else, which covers HTTP, gRPC, and common database clients. Add manual spans around meaningful internal work and propagate context across HTTP headers and queue metadata.

High-cardinality labels like user IDs, raw URLs, or error messages create a separate time series per unique value, which can overwhelm the metrics backend. Keep labels to small fixed sets and put unbounded values in logs and traces instead.

Page only for user-facing symptoms that need immediate action, such as elevated error rate or p99 latency breaching the SLO. Use ticket severity for degradations that can wait, and delete any alert whose correct response is to ignore it.