Generates evaluation suites, scorers, and flag schemas for the Axiom AI SDK.
npx skills add https://github.com/openclaw/clawhub --skill writing-evals
Please help me install this Agent Skill. Skill: writing-evals Source: https://github.com/openclaw/clawhub/tree/main/.agents/skills/writing-evals Command: npx skills add https://github.com/openclaw/clawhub --skill writing-evals
Testing non-deterministic AI features is hard: traditional unit tests cannot verify whether an LLM capability still behaves correctly after a change. This Skill scaffolds complete evaluation suites for the Axiom AI SDK so you can measure correctness, catch regressions, and monitor production quality.
*.eval.ts files with typed scorers, test data, and task functions for classification, retrieval, structured output, and tool-use patterns.createAppScope() with Zod schemas and axiom.config.ts so you can sweep models and parameters via CLI overrides.npx axiom eval.Ask the agent to write evals for your AI feature, for example: "Write evals for the categorize-messages function in src/support-agent.ts using the Axiom AI SDK."
High-intent search queries and answers about installing and using this skill.
Create a `.eval.ts` file that calls `Eval()` with a capability name, a data array of input/expected pairs, a task function invoking your AI code, and at least two scorers. Run it with `npx axiom eval` or `--debug` for local mode.
Classification uses exact match scorers comparing output to expected labels, while retrieval uses set match scorers comparing returned item lists. Structured outputs use field-by-field comparison, and tool use checks expected tool names against actual calls.
Yes, define a flag schema with `createAppScope()` and Zod, then override values from the CLI using `--flag.myCapability.model=gpt-4o-mini`. Every leaf field in the schema must have a `.default()` value, and union types are not supported.
Offline evals run against curated test collections with expected ground truth before deployment. Online evals use `onlineEval` to score live production traffic with reference-free scorers, supporting per-scorer sampling rates and trace linking.
The file must end in `.eval.ts` or `.eval.js` and match the `include` glob in `axiom.config.ts`, which defaults to `**/*.eval.{ts,js}`. Also verify the file contains an `Eval()` call with data, capability, task, and scorers properties.
mattwoodcoSet up OpenTelemetry tracing and structured logging for Next.js applications.
vm0-aiQuery Axiom datasets with time filters and output tabular results.
prashsubAutomate Generative AI agent evaluation with MLflow GenAI and custom scorers.
Accelerated-InnovationPlan LLM evaluation test suites from feature specs and architecture guidance.