Eval Analyzer

What problem does it solve?

Manually sifting through evaluation baseline results to understand AI model performance and identify failure patterns is time-consuming and complex. This Skill automates the analysis, providing actionable insights to improve AI code generation.

Core Features & Use Cases

• Failure Categorization: Automatically classify errors (compile, logic, runtime) and identify top failing benchmarks, helping you focus on critical issues.
• Model Performance Comparison: Compare success rates, costs, and token usage across different AI models, enabling data-driven decisions for model selection.
• Prompt Accuracy Verification: Check for mismatches between prompt documentation and actual language implementation, catching critical prompt bugs before they impact performance.
• Use Case: After running an eval baseline, use this Skill to quickly understand "what's failing" and "why performance dropped," generating a comprehensive report with recommendations for prompt improvements or language fixes.

Quick Start

Get high-level overview of v0.3.16 eval results

.claude/skills/eval-analyzer/scripts/quick_summary.sh eval_results/baselines/v0.3.16

Analyze failures in detail

.claude/skills/eval-analyzer/scripts/analyze_failures.sh eval_results/baselines/v0.3.16

Compare model performance

.claude/skills/eval-analyzer/scripts/compare_models.sh eval_results/baselines/v0.3.16