skills.rest
agenticgogol
agenticgogolCOMMUNITY
Explore all skills by this author →
GitHub

trajectory-eval-new

Evaluates agent tool-use trajectories and attributes failures to the earliest diverging step.

1|9|Updated Jul 11, 2026
Search Tags:#pytest#tool use#trajectory analysis#trace analysis#agent evaluation#transition matrix#failure attribution
One-click install
npx skills add https://github.com/agenticgogol/Edureka_Coding_Agent_Enabled_Demo_11Jul --skill trajectory-eval-new-agenticgogol
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: trajectory-eval-new
Source: https://github.com/agenticgogol/Edureka_Coding_Agent_Enabled_Demo_11Jul/tree/main/.claude/skills/trajectory-eval-new
Command: npx skills add https://github.com/agenticgogol/Edureka_Coding_Agent_Enabled_Demo_11Jul --skill trajectory-eval-new-agenticgogol

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? Final-answer scoring hides where an agent actually went wrong: a trace can reach a correct answer via a bad path, or fail only at the last step after a good run. This Skill scores the trajectory itself and pinpoints the first step where a failed trace diverged from a known-good path. ## Core Features & Use Cases - Three independent code-based checks: pytest-style evaluators over evallib.schema.Trace for tool selection, argument-schema validity, and execution success, each reported separately. - Transition failure matrix: uses scripts/transitions.py to extract per-turn state sequences, build transition counts, and compute each failed trace's earliest divergence index against a reference sequence. - Divergence-ranked reporting: outputs transition_matrix.json and divergence_report.json, ranking which state most often marks the first failure point, cross-referenced against an existing failure taxonomy backlog. - Use Case: After running an agent over evals/traces.jsonl, discover that 12 of 20 failed refund traces first diverge at tool_call:issue_refund because eligibility was never checked — and fix that step first. ## Quick Start Evaluate the traces in evals/traces.jsonl for tool selection, argument validity, and execution success, then build a divergence report showing where failed traces first went wrong.

Frequently Asked Questions about trajectory-eval-new

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate an AI agent's tool-use trajectory instead of its final answer?▼

Write separate code-based checks over each trace: one for tool selection appropriateness, one validating tool arguments against the tool's parameter schema, and one checking tool results for errors. Then compare each failed trace's state sequence against a reference sequence to find the earliest diverging step.

How to find where a failed agent trace first went wrong?▼

Convert each trace into a per-turn state sequence, pick a known-good reference sequence for the same task, and compute the first index where the failed sequence differs. The divergence report maps each trace ID to that earliest divergence index and state.

Why keep tool selection, argument validity, and execution success as separate eval checks?▼

A trace can pass selection and execution while failing argument validity, because a tool endpoint may silently coerce malformed arguments and still report success. Collapsing the three checks into one score hides that failure mode.

What is a transition failure matrix in agent evaluation?▼

It is a count of state-to-state transitions across all trace state sequences, such as tool_call to tool_result. Combined with divergence reports, it ranks which state most often marks the first point of failure, indicating what to fix first.

What are the limitations of divergence-based failure attribution?▼

Position-aligned comparison only detects value differences in the shared prefix; a length mismatch alone is not flagged as divergence. It also requires a reliable reference sequence per task group, which may need manual confirmation when no clean passing trace exists.

Related Skills

Explore More Software Engineering Skills→
riftzen-bitriftzen-bit

skill-comply

Generate compliance specs and classify tool-call traces from claude -p stream-json outputs.

Community
Advanced
MantisWareMantisWare

error-analysis

Analyze LLM traces and logs to categorize failures into a structured error taxonomy.

Community
Advanced
Aradhya0510Aradhya0510
3

agent-evaluation

Evaluate LLM agent output quality using MLflow APIs for datasets and scorers.

Community
Advanced
marchattonmarchatton
5

write-judge-prompt

Design LLM-as-Judge prompts for binary Pass/Fail subjective evaluations.

Community
Intermediate