What problem does it solve? Evaluating LLM outputs at scale is unreliable when judges suffer from position bias, length bias, and inconsistent scoring. This Skill provides production-grade patterns for building LLM-as-judge systems that produce calibrated, reproducible quality assessments. ## Core Features & Use Cases - Direct Scoring & Pairwise Comparison: Implement structured scoring prompts with evidence-based justifications, and pairwise comparison with position-swap protocols to neutralize bias. - Bias Mitigation: Apply concrete techniques for position, length, self-enhancement, verbosity, and authority biases, including cross-model evaluation and length-normalized scoring. - Rubric Generation & Metric Selection: Generate domain-specific scoring rubrics that cut evaluation variance by 40-60%, and select the right statistical metrics (Spearman's rho, Cohen's kappa, F1) to validate against human judgments. - Use Case: When A/B testing two prompt variants, use the pairwise comparison pattern with position swapping to determine a winner with calibrated confidence instead of trusting a single biased judgment. ## Quick Start Use the advanced-evaluation skill to compare these two model responses with position bias mitigation and return a winner with confidence.
