skills.rest
adrianasiam19
adrianasiam19COMMUNITY
Explore all skills by this author →
GitHub

advanced-evaluation

Implements LLM-as-judge evaluation pipelines with bias mitigation, rubrics, and calibrated scoring.

Updated Jan 31, 2026
Search Tags:#llm-as-judge#evaluation#scoring#model evaluation#bias mitigation#pairwise comparison#rubric generation
One-click install
npx skills add https://github.com/adrianasiam19/ATLAS --skill advanced-evaluation-adrianasiam19
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: advanced-evaluation
Source: https://github.com/adrianasiam19/ATLAS/tree/main/.agent/skills/advanced-evaluation
Command: npx skills add https://github.com/adrianasiam19/ATLAS --skill advanced-evaluation-adrianasiam19

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve? Automated evaluation of LLM outputs is unreliable when judges suffer from position bias, length bias, and inconsistent scoring, making it hard to trust quality assessments or compare models fairly. ## Core Features & Use Cases - Direct Scoring and Pairwise Comparison: Score single responses against weighted criteria or compare two responses with position-swapped passes to detect and neutralize position bias. - Rubric Generation: Create domain-specific scoring rubrics with level descriptions, characteristics, and edge-case guidance that reduce evaluation variance by 40-60%. - Bias Mitigation and Metrics: Apply cross-model evaluation, length normalization, and confidence calibration, then validate with Spearman's rho, Cohen's kappa, and position consistency metrics. - Use Case: When A/B testing a new prompt, run pairwise comparisons with position swapping across both prompt versions, then check position consistency and agreement rates to confirm the winner is genuine rather than a bias artifact. ## Quick Start Ask the AI to compare two model responses to a prompt using pairwise comparison with position swapping and return the winner with a calibrated confidence score.

Frequently Asked Questions about advanced-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I implement LLM-as-a-judge evaluation for model outputs?▼

Use direct scoring when objective criteria exist and pairwise comparison for subjective preferences. Structure prompts with explicit criteria, require evidence-based justification before scores, and return structured JSON with scores, justifications, and confidence.

How to mitigate position bias in pairwise LLM comparison?▼

Run the comparison twice with swapped response positions, then check consistency. If both passes agree on the same winner, average the confidences; if they disagree, return TIE with 0.5 confidence since position bias corrupted the result.

When should I use direct scoring vs pairwise comparison?▼

Use direct scoring for objective criteria like factual accuracy or instruction following where ground truth exists. Use pairwise comparison for subjective judgments like tone, style, or persuasiveness, where it achieves higher agreement with human evaluators.

What metrics validate that automated evaluation matches human judgment?▼

Use Spearman's rho or weighted Cohen's kappa for ordinal scales, and agreement rate plus position consistency for pairwise comparisons. Spearman's rho above 0.8 and position consistency above 0.9 indicate a reliable evaluation system.

Why do LLM judges give inconsistent or inflated scores?▼

Common causes include length bias favoring longer responses, missing rubrics causing scale drift, and scoring without justification. Add explicit length-neutrality instructions, generate detailed rubrics, and require evidence before scores.

What are the limitations of LLM-as-judge evaluation?▼

Judges exhibit self-enhancement bias when evaluating their own model's outputs, and minor prompt wording changes can swing scores by 10-20%. Use a different model family for evaluation and version-control evaluation prompts with regression tests.

Related Skills

Explore More Software Engineering Skills→
voidborne-dvoidborne-d

hamel-husain

Build production-grade eval sets with human-validated LLM-as-judge workflows.

Community
Advanced
parhummparhumm
25

detect-dev

Analyze code repositories to produce SARIF-formatted evidence with OpenSSF scoring.

Community
Advanced
gtbaukegtbauke

advanced-evaluation

Implements LLM-as-judge evaluation pipelines with bias mitigation, rubric generation, and metric selection.

Community
Advanced
ContextJet-aiContextJet-ai
29

annotate-traces-for-review

Annotate LLM traces for human review and error analysis.

Official
Intermediate