research.experiment.run

Define an Ablation Matrix with factors and levels in 04_EXPERIMENT_PLAN.md, then this Skill generates the Cartesian product of cells, creates Hydra override configs per cell, and runs each cell across all seeds via uv run python training commands.

Each run gets a run_id combining timestamp, git SHA, and config hash, plus a directory containing config.yaml, metrics.json, events.jsonl, a copied uv.lock, and an auto-generated reproduce.sh script that re-executes the exact run with frozen dependencies.

Yes. Single-GPU environments run cells serially in a loop, while multi-GPU setups assign one GPU per process using CUDA_VISIBLE_DEVICES. Slurm and Ray integration are explicitly out of scope for the current iteration.

Failed runs are never deleted. They are kept with STATUS=failed, the traceback is saved to error.txt, and a POSTMORTEM.md draft is auto-generated in the run directory with hypothesis candidates derived from the event log.

Each line is a JSON event with required fields event, level, ts, run_id, and duration_ms. Error events must also include error_type, error_message, and error_stack, and raw prompt or output text is prohibited for privacy.

It does not support Slurm or Ray cluster scheduling, requires a pre-existing baseline config and experiment plan from earlier phases, and rolls back to Phase 5 if five consecutive cells fail sanity range checks.