skills.rest
navikt
naviktOFFICIAL
Explore all skills by this author →
WebsiteGitHub

observability-debugging

Correlate Mimir metrics, Loki logs, and Tempo traces for incident investigation.

52|13|Updated Jan 10, 2025
Search Tags:#debugging#kubernetes#observability#incident response#loki#tempo#mimir
One-click install
npx skills add https://github.com/navikt/copilot --skill observability-debugging-navikt
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: observability-debugging
Source: https://github.com/navikt/copilot/tree/main/.github/skills/observability-debugging
Command: npx skills add https://github.com/navikt/copilot --skill observability-debugging-navikt

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Troubleshoots production incidents by turning vague symptoms into a structured investigation across observability signals so teams can reach a root cause faster.

Core Features & Use Cases

  • Metrics-first triage with Mimir: Identify error spikes, latency quantiles, saturation, and resource exhaustion signals to determine what to investigate next.
  • Log forensics with Loki: Narrow quickly using indexed labels and structured fields to find failing endpoints, trace_ids, and correlated error context.
  • Trace correlation with Tempo: Locate where time is spent or where failures occur across the call chain to pinpoint bottlenecks and dependency impact.

Quick Start

Ask for a complete debugging path by saying: "My service $APP in $CLUSTER is having high error rate—use Mimir to confirm the spike, Loki to find the trace_id and error context, then Tempo to locate the slow or failing spans."

Frequently Asked Questions about observability-debugging

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I debug production incidents by correlating Kubernetes metrics, logs, and traces?▼

Debug production incidents by querying Mimir metrics to detect error spikes, filtering Loki logs for trace_ids and structured error context, then retrieving Tempo traces to pinpoint slow or failing spans across the call chain.

What is the best way to find a failing endpoint using Loki LogQL in a Kubernetes environment?▼

Find a failing endpoint in Loki by applying label-first filtering and structured LogQL JSON field queries to narrow down indexed labels, isolate trace_ids, and extract correlated error context from your Kubernetes logs.

Can I use Mimir PromQL queries to triage pod resource exhaustion and latency regressions?▼

Use Mimir instant and range PromQL queries to triage pod resource exhaustion, error spikes, and latency quantile regressions to determine which Kubernetes service requires deeper investigation.

How do I locate slow spans in a distributed trace using Tempo?▼

Locate slow spans in Tempo by executing trace search queries or retrieving traces directly by trace_id to visualize where time is spent, identify bottlenecks, and assess dependency impact across the call chain.

Do I need tenant or org headers to query Mimir, Loki, and Tempo for incident response?▼

Querying Mimir, Loki, and Tempo for incident response requires passing tenant or org headers to authenticate and scope your PromQL, LogQL, and trace retrieval requests to the correct Kubernetes-based environment.

Related Skills

Explore More Software Engineering Skills→
williamzujkowskiwilliamzujkowski
6

Observability Stack Configurator

Generate Prometheus, OpenTelemetry, Grafana, and CloudWatch configurations for Kubernetes and cloud deployments.

Community
Advanced
openeuler-mirroropeneuler-mirror
3

fault-rca-report-generation

Generate structured root cause analysis reports from system logs and performance metrics.

Official
Advanced
masermediagroup-stackmasermediagroup-stack
1

implementing-observability

Implement OpenTelemetry metrics, logs, and traces with LGTM stack deployment.

Community
Advanced
RepairYourTechRepairYourTech
6

datadog

Configure Datadog for APM, log management, metrics, dashboards, and alerting.

Community
Advanced