observability

Write logs as events with a stable name and variable fields instead of prose sentences. For example, log "upload_failed" with fields like user_id and duration_ms, so you can filter, group, and aggregate on any field later.

Use RED for request-serving code (rate, errors, duration) and USE for resources (utilization, saturation, errors). Always use histograms for latency, express errors as ratios, and instrument every outbound dependency with its own duration and error rate.

A metric's cost is the product of its label values, so adding a label like user_id can multiply thousands of series into millions and cause an outage. Keep high-cardinality identifiers like user ids and request ids in logs and traces instead.

Alert on metrics, because a reworded log message silently disables a log-based alert. Alert on user-visible symptoms such as SLO burn rate and p99 latency, then diagnose with traces and confirm with logs.

Never log passwords, tokens, session ids, API keys, card numbers, government ids, Authorization headers, or full personal records. Log a reference such as a user id or hash, and enforce redaction at the logger rather than at each call site.