What problem does it solve? Automated evaluation of LLM outputs is unreliable when judges suffer from position bias, length bias, and inconsistent scoring. This Skill provides production-grade patterns for building LLM-as-judge systems that correlate with human judgment. ## Core Features & Use Cases - Direct Scoring & Pairwise Comparison: Choose the right evaluation approach based on whether objective ground truth exists, with structured prompt templates for each. - Bias Mitigation: Apply position swapping, length normalization, cross-model evaluation, and evidence requirements to counter systematic judge biases. - Rubric Generation & Metric Selection: Generate domain-specific scoring rubrics that reduce evaluation variance by 40-60%, and select appropriate statistical metrics (Spearman's rho, Cohen's kappa, F1) to validate against human judgments. - Use Case: When A/B testing two prompt variants, run pairwise comparisons with position swapping to get a consistent winner with calibrated confidence instead of a single biased judgment. ## Quick Start Ask the AI to compare two model responses to a prompt using pairwise comparison with position-swap bias mitigation and return a winner with confidence score.
