skills.rest
Syedyasir001
Syedyasir001COMMUNITY
Explore all skills by this author →
GitHub

advanced-evaluation

Implement LLM-as-judge pipelines with direct scoring, pairwise comparison, and bias mitigation.

Updated Apr 17, 2026
Search Tags:#llm-as-judge#evaluation#model evaluation#bias mitigation#pairwise comparison#rubric generation#direct scoring
One-click install
npx skills add https://github.com/Syedyasir001/RVULibPass --skill advanced-evaluation-syedyasir001
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: advanced-evaluation
Source: https://github.com/Syedyasir001/RVULibPass/tree/main/.agent/skills/library/advanced-evaluation
Command: npx skills add https://github.com/Syedyasir001/RVULibPass --skill advanced-evaluation-syedyasir001

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve? Automated evaluation of LLM outputs is unreliable when judges suffer from position bias, length bias, and inconsistent scoring, making it hard to trust quality assessments or compare models fairly. ## Core Features & Use Cases - Direct Scoring & Pairwise Comparison: Build structured evaluation prompts with criteria, calibrated scales, and position-swapped comparisons for subjective judgments. - Bias Mitigation: Apply concrete techniques against position, length, self-enhancement, verbosity, and authority biases, including swap protocols and length-normalized scoring. - Rubric Generation & Metric Selection: Generate domain-specific scoring rubrics and choose the right agreement metrics (Spearman's rho, Cohen's kappa, F1) to validate against human judgment. - Use Case: When A/B testing two prompt variants, use the pairwise comparison pattern with position swapping to determine which variant produces better responses, with calibrated confidence scores. ## Quick Start Ask the AI to compare two model responses to a prompt using pairwise comparison with position bias mitigation and return a winner with confidence.

Frequently Asked Questions about advanced-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I implement LLM-as-a-judge evaluation for model outputs?▼

Use direct scoring when objective criteria exist, or pairwise comparison for subjective preferences. Structure prompts with explicit criteria, require justification before scores, and return structured JSON with evidence and confidence.

How to mitigate position bias in pairwise LLM comparison?▼

Run the comparison twice with swapped response positions, then check consistency. If both passes agree, average the confidences; if they disagree, return TIE with 0.5 confidence since position bias influenced the result.

When should I use direct scoring vs pairwise comparison?▼

Use direct scoring for objective criteria like factual accuracy or instruction following where ground truth exists. Use pairwise comparison for subjective judgments like tone, style, or persuasiveness, where it achieves higher human-judge agreement.

What metrics validate automated evaluation against human judgment?▼

Use Spearman's rho or Kendall's tau for ordinal scales, Cohen's kappa for categorical judgments, and precision/recall/F1 for binary tasks. Track position consistency and length-score correlation as bias indicators.

Why do LLM judges give inconsistent scores for the same response?▼

Inconsistency usually comes from missing rubrics, overloaded criteria measuring multiple things, or evaluation prompt sensitivity where minor wording changes cause score swings. Generate detailed rubrics with level descriptions and edge cases to reduce variance by 40-60%.

What are the limitations of LLM-as-judge evaluation?▼

LLM judges exhibit systematic biases including position, length, self-enhancement, verbosity, and authority bias. They require bias mitigation protocols, calibration against human judgments, and periodic rubric reviews to remain reliable.

Related Skills

Explore More Software Engineering Skills→
mindmorassmindmorass
2

embedding-comparison

Benchmark embedding models on document, code, and QA datasets.

Community
Intermediate
curiositechcuriositech
10

dag-confidence-scorer

Score DAG outputs with calibrated multi-factor confidence metrics.

Official
Advanced
gtbaukegtbauke

advanced-evaluation

Implements LLM-as-judge evaluation pipelines with bias mitigation, rubric generation, and metric selection.

Community
Advanced
AI-Foundry-CoreAI-Foundry-Core
4

llm-evaluation

Automate LLM evaluation with BLEU, ROUGE, BERTScore, and human feedback.

Official
Advanced