rc-observability

Emit one JSON object per line with timestamp, level, service name, correlation_id, message, and relevant identifiers like order_id or retry_count. Never log PII, secrets, tokens, or full customer payloads, and use log levels with discipline: error for actionable failures, warn for degraded states, info for business milestones.

Generate the correlation ID at the edge (API Gateway or first handler) if not provided upstream, then propagate it in SQS MessageAttributes or a dedicated traceContext field in EventBridge events. Every log from every service in the chain must include the same correlation_id to reconstruct a message's path.

RED metrics (Rate, Errors, Duration) measure service health from the consumer's perspective and apply to endpoints and handlers. USE metrics (Utilization, Saturation, Errors) measure finite resources like queues, connection pools, and Lambda memory to reveal what saturates first under load.

Alert on user-facing symptoms like error rate or DLQ growth, not internal causes like high CPU. Derive thresholds from error budget burn rate rather than arbitrary limits, link a runbook in every critical alert, and delete or fix any alert that is repeatedly ignored.

Do not use it for local debugging (use a diagnosis skill instead), code profiling, or configuring a specific APM vendor. It focuses on instrumentation design, alerting strategy, and incident analysis for services already in or heading to production.

Classify severity by real user impact, contain and communicate first, then apply 5-whys until reaching a systemic cause rather than a local symptom. Document timeline, impact, root cause, and action items with owners and deadlines, focusing on process gaps rather than individual mistakes.