performance-profiling

Record a baseline first: capture p50/p95/p99 latency, throughput, error rate, and CPU/memory under a repeatable workload. Then profile the hot path with language-native profilers or flame graphs, test one narrow hypothesis, and re-measure before merging any change.

Use eBPF-based zero-instrumentation profilers such as Pyroscope or the OpenTelemetry Profiling SIG for always-on profiling. Keep overhead under 1.5% CPU with bounded memory, get explicit approval, and ensure profiling endpoints are access-controlled.

Measure model latency p50/p95/p99 separately from service latency, since LLM calls often dominate. For GPU services like vLLM or Ollama, profile GPU utilization, VRAM, KV cache hit rate, and time-to-first-token using nvitop or PyTorch Profiler.

The mean can improve while tail latency worsens, hiding user-facing regressions. Always report p50, p95, and p99 before and after a change, and treat tail regressions as release-blocking issues.

Avoid production profiling without explicit approval and a safety plan. Use the least invasive method, keep duration short, enforce the 1.5% CPU overhead limit, and coordinate with service owners when the workload is customer-facing.