observability-patterns

Structured logging emits each entry as a JSON object with required fields: timestamp, level, message, service, and request_id. Use snake_case event names like user_login and explicitly select fields to log rather than interpolating strings.

Use ERROR only for unrecoverable failures requiring human action, WARN for degraded conditions, INFO for significant business events, and DEBUG for troubleshooting detail. Validation failures and 404s are DEBUG, not ERROR.

Use an allowlist approach: explicitly select non-sensitive fields instead of dumping request bodies. Mask identifiers when needed, such as logging only a key prefix or card last four digits, and never log passwords, tokens, or card numbers.

Generate or accept an X-Request-ID header at the entry point, store it in async local storage, and include it in every log entry. Propagate the same ID in headers to downstream services and into background job payloads.

Labels like user_id or request_id create millions of distinct time series, overwhelming the metrics backend. Use labels only for bounded sets such as status codes, endpoints, and service names.

Every alert needs a duration filter (for example 5 minutes) to avoid firing on spikes, a severity, an owner, and a runbook with concrete investigation steps. More than two or three actionable alerts per week per service indicates wrong thresholds.