anthropic-evaluations

What problem does it solve?

This skill provides a framework for designing and evaluating AI agent tasks using Anthropic-style eval patterns. It helps teams define standardized graders, templates, and evaluation harnesses to ensure robust QA across coding, conversational, and research agents.

Core Features & Use Cases

• Grader types including code-based, model-based, and human graders.
• Templates and references: YAML templates for coding and conversational evals, plus reference materials.
• Use cases span building evaluation suites, benchmarking agent capabilities, and QA for research agents.

Quick Start

Review the references/ directory for evaluation patterns and starter templates such as coding-agent-eval.yaml or conversational-agent-eval.yaml. Adapt a template to your task, configure your graders and tracked metrics, and run your evaluation harness to collect transcripts, tool usage, and pass/fail signals.