pairwise-compare-evals

What problem does it solve?

Pairwise-compare-evals provides a structured method to rank AI safety evaluations by systematically contrasting each eval across a fixed set of criteria using the Saaty scale, producing a transparent justification trail.

Core Features & Use Cases

• Compares all assessed evals head-to-head on 10 criteria (7 rubric dimensions + 3 porting criteria) to generate per-pair scores and overall rankings.
• Generates batch prompts and aggregates results into a matrix and summary rankings, enabling data-driven prioritization of evals.
• Use Case: run the full evaluation comparison workflow to surface top-performing evals and identify gaps in coverage or feasibility.

Quick Start

Run the full pairwise evaluation workflow to generate batches and start processing.