add-benchmark

Define a Benchmark value with Dataset, Solver, and Scorer layers, add its Zod config schema and metadata, register it in registry.ts, and wire the CLI dispatch. Finish with colocated tests and a smoke run compared against a published target score.

Use makeHfDatasetLayer, which pages the HuggingFace rows endpoint and exposes results as a Stream of Samples so memory stays flat. Write a pure, deterministic recordToSample that validates every field and throws typed errors on surprises.

Identical completions across epochs indicate cache salting failure. Response caching is keyed by session id, epoch, retry attempt, and per-call salt, so solvers issuing multiple calls per sample must wrap each in withCallCacheSalt.

Reuse extractMcqAnswer, which handles bold, LaTeX, boxed, and non-Latin letter forms, and shuffle options with seededPermutation keyed on sample index. Always record the extracted answer so failed extraction is distinguishable from a wrong answer.

Prefer deterministic string matching or sandbox state inspection first, using an LLM judge only when the task requires it. Pin the judge model, temperature, and reasoning effort, and measure judge run-to-run variance before attributing score deltas to the model under test.

The same model on two providers can score differently due to provider variance. Pin providerOnly or endpointId for any comparison you publish, and verify a pinned run actually reaches the pinned endpoint via the X-OR-Endpoint-Id header.