observability-fundamentals

Observability lets you ask arbitrary questions not defined ahead of time by exploring high-cardinality event data, while monitoring relies on pre-defined dashboards and alerts. Observability decides data shape at query time; monitoring decides it at instrumentation time.

A wide event is a flat key-value record capturing the full context of a unit of work, such as user ID, endpoint, duration, and error status. In OpenTelemetry, a span is a wide event, and every attribute becomes a queryable dimension.

Each unique label value creates a new time series, so 100,000 users across 10 metrics produces one million time series. This cardinality explosion forces metrics systems to keep labels low-cardinality, stripping away debugging context.

The core analysis loop is Define, Visualize, Investigate, Evaluate. You frame a question, run a query to see the problem's shape, narrow down with BubbleUp and traces, then confirm the hypothesis by querying with and without the suspected cause.

Use structured events when you need aggregate queries like P99 latency, automatic outlier detection via BubbleUp, or cross-service correlation through trace context. Logs require expensive full scans for aggregation and manual correlation IDs across services.