What problem does it solve? Open-ended ML experimentation often wastes compute on unvetted ideas, and single-hypothesis loops never compare alternatives before spending a training run. This Skill replaces the hypothesis step with an idea tournament: multiple research agents propose competing architecture changes, a self-calibrating Judge critiques and ranks them against a rubric, and only the winning change is run and measured. ## Core Features & Use Cases - Adversarial idea tournament: Spawn n ResearchAgent proposers per iteration, gate ideas on falsifiable predictions, score them on grounding/impact/feasibility, and rank via de-biased pairwise comparison. - Self-calibrating Judge: The Judge logs predicted vs realized metric deltas in a calibration ledger, tracks its selection hit-rate, and refines its working rubric's weights and anchors from realized outcomes. - Full experiment mechanics: Snapshot or branch isolation, budgeted runs, mandatory analysis plans with real artifacts, keep/revert decisions, and append-only ledgers (results.tsv, calibration.tsv, judge_lessons.md). - Use Case: Point the loop at a training repo (e.g., an OCR recognizer with a train.py entrypoint), set val accuracy as the metric, and let it iteratively propose, vet, run, and analyze one architecture change per iteration until interrupted. ## Quick Start Ask the agent to start the tournament autoresearch loop on this repository with validation accuracy as the metric and the training script as the run command.
