implementing-observability

Install opentelemetry-sdk, opentelemetry-exporter-otlp, and opentelemetry-instrumentation-fastapi, then call FastAPIInstrumentor.instrument_app(app) after configuring a TracerProvider with an OTLP exporter. The setup_otel.py script generates this configuration automatically.

LGTM combines Loki for logs, Grafana for visualization, Tempo for traces, and Mimir for metrics into a self-hosted observability platform. Grafana Alloy collects OTLP telemetry and routes each signal type to its backend.

Inject trace_id and span_id into every structured log record using the current span context from OpenTelemetry. Then query Loki with the trace_id value to find all logs belonging to a specific trace.

Yes, OpenTelemetry provides SDKs for Rust (opentelemetry with tracing-opentelemetry) and Go (go.opentelemetry.io/otel). Both support OTLP export and framework auto-instrumentation for Axum, Actix-web, Gin, and net/http.

Verify the OTLP endpoint is reachable on port 4317, confirm Grafana Alloy is running and receiving data, and check application logs for exporter errors. Also ensure the service.name resource attribute matches your Grafana query.

Skip it for proof-of-concept projects without production deployment or systems handling under 100 requests per day, where console logging is sufficient. The overhead of SDK setup and stack hosting outweighs the benefit at that scale.