skills.rest
corticalstack
corticalstackCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

advanced-evaluation

Implements LLM-as-judge evaluation pipelines with bias mitigation, rubrics, and calibrated scoring.

1|1|Updated Feb 2, 2026
Search Tags:#llm-as-judge#evaluation#scoring#model evaluation#bias mitigation#pairwise comparison#rubric generation
One-click install
npx skills add https://github.com/corticalstack/flow --skill advanced-evaluation-corticalstack
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: advanced-evaluation
Source: https://github.com/corticalstack/flow/tree/main/.claude/skills/llm-as-judge
Command: npx skills add https://github.com/corticalstack/flow --skill advanced-evaluation-corticalstack

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve? Automated evaluation of LLM outputs is unreliable when judges suffer from position bias, length bias, and inconsistent scoring, making it hard to trust quality assessments or compare model outputs at scale. ## Core Features & Use Cases - Direct Scoring and Pairwise Comparison: Choose the right evaluation approach based on whether objective ground truth exists, with structured prompts requiring evidence-based justification before scores. - Bias Mitigation Protocols: Apply position swapping, length normalization, cross-model evaluation, and fact-checking layers to counter position, length, self-enhancement, verbosity, and authority biases. - Rubric Generation and Metric Selection: Generate domain-specific scoring rubrics that cut evaluation variance by 40-60%, and select appropriate metrics (Spearman's rho, Cohen's kappa, F1) to validate against human judgments. - Use Case: When A/B testing a new prompt, run pairwise comparisons with position swapping across both prompt versions, then validate judge reliability against human spot-checks using Spearman correlation before rolling out the winner. ## Quick Start Ask the agent to compare two model responses for a prompt using pairwise comparison with position bias mitigation and return a calibrated confidence score.

Frequently Asked Questions about advanced-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I implement LLM-as-a-judge evaluation for model outputs?▼

Choose direct scoring when objective ground truth exists (factual accuracy, instruction following) or pairwise comparison for subjective preferences (tone, style). Structure prompts to require evidence-based justification before scores, which improves reliability by 15-25%.

How to mitigate position bias in pairwise LLM comparison?▼

Run every comparison twice with swapped response positions, then check consistency. If both passes agree, average the confidences; if they disagree, return TIE with 0.5 confidence since position bias corrupted the result.

What metrics should I use to validate LLM judges against human evaluation?▼

Use Spearman's rho or Kendall's tau for ordinal scales, Cohen's kappa for categorical judgments, and precision/recall/F1 for binary tasks. Target Spearman's rho above 0.8 and position consistency above 90% for a trustworthy judge.

When should I use direct scoring vs pairwise comparison?▼

Use direct scoring for objective criteria like factual accuracy or format compliance where ground truth exists. Use pairwise comparison for subjective judgments like tone or persuasiveness, where it achieves higher agreement with human evaluators.

Why do LLM judges give inconsistent or inflated scores?▼

Common causes include length bias (longer responses score higher), self-enhancement bias (models favor their own outputs), and missing rubrics. Mitigate with explicit anti-length instructions, cross-model evaluation, and detailed per-level scoring rubrics.

How do I scale LLM evaluation for high-volume production use?▼

Apply hierarchical evaluation with a cheap model for screening and an expensive model for edge cases, use a panel of multiple LLM judges for high-stakes decisions, or route low-confidence cases to human review with feedback loops.

Related Skills

Explore More Software Engineering Skills→
mindmorassmindmorass
2

embedding-comparison

Benchmark embedding models on document, code, and QA datasets.

Community
Intermediate
langwatchlangwatch
3.5k

experiments

Creates and runs LangWatch batch experiments to test agents against datasets with evaluators.

Official
Advanced
Miosa-osaMiosa-osa
226

validate-evaluator

Calibrate LLM evaluators against human-labeled datasets and generate Markdown calibration reports.

Official
Advanced
AI-Foundry-CoreAI-Foundry-Core
4

llm-evaluation

Automate LLM evaluation with BLEU, ROUGE, BERTScore, and human feedback.

Official
Advanced