compare-compositions

Compares composition methods on a quality-cost frontier and writes composition-profile.json.

Updated Jun 20, 2026
One-click install
npx skills add https://github.com/ensj/skill-eval-framework --skill compare-compositions-ensj
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: compare-compositions
Source: https://github.com/ensj/skill-eval-framework/tree/main/.claude/skills/compare-compositions
Command: npx skills add https://github.com/ensj/skill-eval-framework --skill compare-compositions-ensj

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Choosing between composition topologies (e.g. deep vs wide research orchestration) over the same atomic skills is guesswork without measured data. This Skill runs each method over a shared task set and seeds, judges outputs against a contract rubric, measures cost, and produces a Pareto frontier with a statistically grounded winner. ## Core Features & Use Cases - Multi-method benchmarking: Runs each composition method end-to-end across tasks and seeds via the run-composition skill, recording agent and token costs per run. - Rubric-based LLM judging: Dispatches a judge agent per output with the contract rubric, computing the overall quality score in the harness as the unweighted mean of accuracy, coverage, citations, and freshness subscores. - Pareto frontier analysis: Aggregates quality and cost per method, then uses the compose-eval CLI to compute the Pareto set and winner, writing composition-profile.json. - Use Case: Compare a wide-fanout research orchestration against a lean-linear pipeline on the same research questions to find which delivers the best quality per token budget. ## Quick Start Compare the research-orchestration and research-lean-linear composition methods on the research contract and show me the quality-cost frontier.

Frequently Asked Questions about compare-compositions

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I compare composition methods in skill-eval-framework?▼

Run each method over a shared task set and seeds via the run-composition skill, judge each output against the contract rubric, then aggregate quality and cost per method. The compose-eval CLI computes the Pareto set and winner into composition-profile.json.

How is the quality score computed for composition outputs?▼

A judge agent returns subscores for accuracy, coverage, citations, and freshness. The harness computes overall as the unweighted mean of those four subscores; judge-computed overall scores are never accepted because LLMs should not do numeric aggregation.

What prerequisites are needed before comparing compositions?▼

You need npm run build to emit patterns/<method>.js and dist/compose-eval/cli.js, plus a winner.json for each atomic step from running eval-sweep. Without these, run-composition cannot resolve prompts.

Can I compare old and new composition profiles directly?▼

No. Profiles written before hardening-3 carry judge-computed quality_mean values that run about 0.10 harsher than the subscore mean. Re-run all methods under the new harness-computed derivation before cross-comparing.

Why does composition comparison use multiple seeds?▼

The judge is an LLM, so quality scores carry variance across runs. Multiple seeds per task let the skill report quality_stdev alongside quality_mean, making method differences distinguishable from judge noise.