eval-running

What problem does it solve?

This Skill automates the process of evaluating and benchmarking the quality of Loa skills, ensuring that framework changes do not introduce regressions and that skills maintain their expected performance.

Core Features & Use Cases

• Run Evaluation Suites: Execute predefined suites like 'framework' or 'regression' to assess skill behavior.
• Target Specific Tasks or Skills: Focus evaluations on individual tasks or all tasks associated with a particular skill.
• Update Baselines: Easily update existing evaluation baselines with new results, providing a rationale for the changes.
• Use Case: After deploying a new version of the Loa framework, run the 'regression' suite to confirm that existing skills still function as expected and haven't been negatively impacted.

Quick Start

Run the framework correctness evaluation suite by typing /eval --suite framework.