otel-instrumentation

Set three environment variables: OTEL_EXPORTER_OTLP_ENDPOINT (https://api.honeycomb.io), OTEL_EXPORTER_OTLP_HEADERS with your API key, and OTEL_SERVICE_NAME. Then install the OTel SDK and exporter for your language and add auto-instrumentation for your frameworks.

Use programmatic SDK setup rather than the opentelemetry-instrument CLI, which does not integrate cleanly with ASGI lifespans. Create a telemetry module with a TracerProvider and BatchSpanProcessor, then call FastAPIInstrumentor.instrument_app(app) after all routes are mounted.

Head sampling is simpler but can miss rare errors; a 0.1% error at 1% sampling appears once per 100,000 requests. Tail sampling via Honeycomb Refinery or the Collector keeps errors and slow traces while sampling routine traffic, at the cost of extra infrastructure.

SQLAlchemyInstrumentor does not support async engines directly and raises NotImplementedError at startup. Pass the underlying sync engine instead: SQLAlchemyInstrumentor().instrument(engine=async_engine.sync_engine).

Lambda freezes the process when the handler returns, so BatchSpanProcessor's background flush never fires and queued spans are dropped. Call provider.forceFlush() in a finally block before the handler returns, or use the AWS managed OTel layer which exports after the response.

Create a span only if the operation meaningfully impacts latency or failures and grouping it by name produces useful trends. For sub-operation timings, prefer attributes on the parent span, which are easier to query and work directly with BubbleUp.