advanced-evaluation

Build a pipeline with a criteria loader, primary scorer, bias mitigation layer, and confidence scoring. Use direct scoring for objective criteria like factual accuracy, and pairwise comparison for subjective preferences like tone and style.

Evaluate twice with swapped response positions, then check consistency. If both passes agree on the winner, average the confidences; if they disagree, return TIE with 0.5 confidence since position bias corrupted the result.

Use direct scoring when objective ground truth exists, such as factual accuracy or instruction following. Use pairwise comparison for subjective judgments like tone, style, or persuasiveness, where it achieves higher human-judge agreement.

Use Spearman's rho or Kendall's tau for ordinal scales, Cohen's kappa for categorical judgments, and agreement rate for pairwise decisions. Values above 0.8 for Spearman's rho indicate strong correlation with human evaluators.

Inconsistency stems from position bias, length bias, missing rubrics, and prompt sensitivity where minor wording changes cause 10-20% score swings. Mitigate with position swapping, explicit length-neutrality instructions, detailed rubrics, and version-controlled prompts.

LLM judges exhibit self-enhancement bias toward their own outputs, authority bias toward confident tone, and verbosity bias toward longer responses. Automated evaluation is only valuable when validated against human judgments with correlation monitoring.