Skill Evals

Define per-skill assertions in evals.json covering output file patterns, minimum word counts, required and forbidden regex patterns, and numeric ranges. The eval workflow checks the latest output of each skill and diffs results against the prior run to flag new failures.

Compare each run's results table against the most recent prior eval article, classifying skills as NEW_FAIL, FIXED, STILL_FAIL, or STABLE. A precedence-based verdict such as SKILL_EVALS_REGRESSED summarizes whether action is needed.

Each evals.json entry supports output_pattern glob, min_words, required_patterns, forbidden_patterns, and numeric_checks with min/max bounds. Optional fields like skip_if_not_found and expected_cadence extend the schema without breaking existing entries.

Yes. Before filing, it scans memory/issues/INDEX.md and skips creating an issue if an open one already names the failing skill. It also auto-closes only issues it filed itself, marked detected_by: skill-evals, when a skill recovers.

Notifications are gated: they fire only on regression, quality drop, coverage cliff, or recovery verdicts. A fully green SKILL_EVALS_OK run writes the article and log entry silently, so a notification always signals action is needed.

Skills missing from evals.json surface as coverage gaps with inferred patterns via the eval-audit script. Skills with no matching output file are marked NO_OUTPUT with root cause no_file_match, and stale files older than twice their expected cadence are flagged STALE.