inspect-a-trace

Correlate agent runs across Postgres checkpoints, Loki, Tempo, and Grafana to reconstruct failures.

1|Updated Aug 14, 2026
One-click install
npx skills add https://github.com/zhiyuan-zhang0206/Ava --skill inspect-a-trace-zhiyuan-zhang0206
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: inspect-a-trace
Source: https://github.com/zhiyuan-zhang0206/Ava/tree/main/.agents/skills/inspect-a-trace
Command: npx skills add https://github.com/zhiyuan-zhang0206/Ava --skill inspect-a-trace-zhiyuan-zhang0206

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve? When an AI agent misbehaves, the evidence is scattered across four live systems — Postgres checkpoints hold the conversation, Loki holds the event stream, Tempo holds the span tree, and Grafana holds the browsable view. This Skill provides the correlation know-how to go from a symptom like "agent 3048 behaved badly around 14:00" to the exact conversation, events, spans, and a shareable trace link. ## Core Features & Use Cases - Run discovery: Narrow to one agent's run by agent_id and time window in Loki, or query Postgres checkpoints for conversation content and history across compaction segments. - Trace toolchain: Three chained scripts (fetch_trace.py, read_trace.py, render_trace.py) find traces via TraceQL, fetch full span sets from the local mirror or Tempo, and render a self-contained HTML waterfall report. - Failure diagnosis recipes: Ready-made LogQL queries for exec failures, LLM spend, turn outcomes, halt reasons, and crash loops. - Use Case: A user reports an agent looped and burned tokens yesterday. You query Loki for the agent's event river, extract the trace_id of the bad turn, fetch its spans from the mirror, join the checkpoint messages via the gateway, and hand the user a Grafana Explore link. ## Quick Start Ask the assistant to inspect what agent 3048 did around 14:00 yesterday and produce a trace report with a Grafana link.

Frequently Asked Questions about inspect-a-trace

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I trace what an AI agent did during a specific run?▼

Start in Loki with the agent's event stream using {service_name="unknown_service", agent_id="<id>"} | json, narrow to a time window, and extract a trace_id. Then fetch that trace's spans from the local mirror or Tempo and join the checkpoint messages for the conversation content.

How to correlate a trace_id with agent conversation checkpoints?▼

Query the checkpoints table with metadata->>'trace_id' to find the checkpoint a turn committed, or use the gateway endpoint GET /api/agents/{agent_id}/traces/{trace_id}/messages. The read_trace.py --with-content flag performs this join automatically.

Why does fetching a trace from Tempo fail with a size error?▼

Tempo's full-trace API refuses traces above its 5 MB cap. Use the local mirror source instead — fetch_trace.py --source mirror scans $AVA_HOME/traces/spans*.jsonl files, which are complete, uncapped, and need no network access.

Can I get prompt and completion content from Tempo spans?▼

No. Spans are metadata-only; prompts, completions, and tool arguments are stripped at record time. Conversation content comes from Postgres checkpoints via shared.checkpoint loaders or the gateway messages endpoint, keyed by trace_id.

What are the limitations of Loki retention for old agent runs?▼

Loki retains events for 168 hours, so runs older than a week are gone from the event stream. Older history survives only as compaction-boundary checkpoint snapshots in Postgres, which the reaper deliberately preserves.