skills.rest
ludotype
ludotypeCOMMUNITY
Explore all skills by this author →
GitHub

advanced-evaluation

Implements LLM-as-judge evaluation pipelines with bias mitigation, rubrics, and calibrated scoring.

1|Updated May 16, 2026
Search Tags:#llm-as-judge#evaluation#scoring#model evaluation#bias mitigation#pairwise comparison#rubric generation
One-click install
npx skills add https://github.com/ludotype/game-adventurertown --skill advanced-evaluation-ludotype
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: advanced-evaluation
Source: https://github.com/ludotype/game-adventurertown/tree/main/.agent/skills/advanced-evaluation
Command: npx skills add https://github.com/ludotype/game-adventurertown --skill advanced-evaluation-ludotype

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve? Automated evaluation of LLM outputs is unreliable when judges suffer from position bias, length bias, and inconsistent scoring. This Skill provides production-grade patterns for building LLM-as-judge systems that produce consistent, calibrated, and human-correlated quality assessments. ## Core Features & Use Cases - Direct Scoring & Pairwise Comparison: Implement single-response scoring for objective criteria and pairwise comparison with position-swap mitigation for subjective preferences. - Bias Mitigation: Apply concrete techniques against position, length, self-enhancement, verbosity, and authority biases, including swap protocols and length-normalized scoring. - Rubric Generation: Generate domain-specific scoring rubrics with level descriptions, edge cases, and strictness calibration to reduce evaluation variance by 40-60%. - Use Case: When A/B testing two prompt variants, use the pairwise comparison pattern with position swapping to determine which variant produces better responses, with confidence calibrated to position consistency. ## Quick Start Ask the AI to compare two model responses to a prompt using pairwise comparison with position-swap bias mitigation and return a winner with calibrated confidence.

Frequently Asked Questions about advanced-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I implement LLM-as-a-judge evaluation for model outputs?▼

Use direct scoring when objective criteria exist and pairwise comparison for subjective preferences. Structure prompts with clear criteria, require evidence-based justification before scores, and return structured JSON with scores, justifications, and confidence.

How to mitigate position bias in pairwise LLM comparison?▼

Evaluate twice with swapped response positions, then check consistency. If both passes agree on the winner, average the confidences; if they disagree, return TIE with 0.5 confidence since position bias corrupted the result.

When should I use direct scoring vs pairwise comparison?▼

Use direct scoring when objective ground truth exists, such as factual accuracy or instruction following. Use pairwise comparison for subjective judgments like tone, style, or persuasiveness, where it achieves higher human-judge agreement.

What metrics validate automated evaluation against human judgments?▼

Use Spearman's rho or Kendall's tau for ordinal scales, Cohen's kappa for categorical judgments, and agreement rate plus position consistency for pairwise comparisons. Target Spearman's rho above 0.8 and position consistency above 0.9.

Why do LLM judges give inconsistent evaluation scores?▼

Inconsistency stems from position bias, length bias, missing rubrics, and prompt sensitivity where minor wording changes cause 10-20% score swings. Mitigate with position swapping, explicit length-neutrality instructions, detailed rubrics, and version-controlled evaluation prompts.

What are the limitations of LLM-as-judge evaluation?▼

LLM judges exhibit self-enhancement bias toward their own outputs, authority bias toward confident tone, and verbosity bias toward longer responses. They require validation against human judgments and periodic rubric recalibration to remain reliable.

Related Skills

Explore More Software Engineering Skills→
agenticgogolagenticgogol
1

production-monitor-setup

Generates a production monitoring configuration for deployed agents using tracing tools and calibrated judges.

Community
Intermediate
curiositechcuriositech
10

dag-confidence-scorer

Score DAG outputs with calibrated multi-factor confidence metrics.

Official
Advanced
thejefflarsonthejefflarson
20

overreliance

Gate LLM outputs by confidence and route high-stakes cases to human review.

Community
Advanced
ollieb89ollieb89

llm-evaluation

Evaluate LLM outputs with automated metrics and LLM-as-Judge workflows.

Community
Advanced