excrtx-quality-skilljudge

Run the skill_judge.py script with --all --d1-only for deterministic structural checks, then use the judge prompt template for LLM-based evaluation of clarity, alignment, fitness, and token economy. Results produce categorical labels and an overall PASS, IMPROVE, or REWRITE verdict.

LLM-as-Judge uses a language model to score artifacts against a structured rubric with categorical labels and mandatory chain-of-thought reasoning. It avoids numeric 1-10 scales because LLMs cluster around 7-8, making categorical verdicts more reliable and actionable.

Save sweep results as a JSON baseline, then run the script with --compare-baseline pointing to that file. Any skill whose verdict regresses from PASS to IMPROVE or worse fails the pipeline with a non-zero exit code.

Using the same model for writing and judging causes self-enhancement bias that inflates scores. Use a different model for judging, or calibrate the rubric with human-graded examples to correct leniency drift.

Do not use it for evaluating agent runtime behavior, which requires session traces, or for LLM output quality, which needs output-level review. It also does not replace code review or linters for source code quality.