reproducibility

Establishes reproducible experiment records covering seeds, configs, environments, logs, and artifacts.

1|Updated Aug 14, 2026
One-click install
npx skills add https://github.com/zhiyuan-zhang0206/Ava --skill reproducibility-zhiyuan-zhang0206
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: reproducibility
Source: https://github.com/zhiyuan-zhang0206/Ava/tree/main/ava_builtins/skills/ava-serious-research/principles/reproducibility
Command: npx skills add https://github.com/zhiyuan-zhang0206/Ava --skill reproducibility-zhiyuan-zhang0206

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Experimental results often cannot be re-derived because seeds, configs, environments, and logs were never recorded at run time, making every number untrustworthy and every conclusion unverifiable. ## Core Features & Use Cases - Day-one record template: Creates an experiment directory with config.yaml, seeds.txt, env.lock, run.sh, logs/, and outputs/ before any run starts. - Calibration checkpoint: Reproduces a known published result before trusting any new measurement, validating the pipeline as an instrument. - Sealed test set: Splits and hash-archives the test set before preprocessing so it participates in no computation until final evaluation. - Full trace archiving: Stores every run's command, stdout/stderr, random seed, and git commit together with results. - Use Case: Before launching a new model training run, use this Skill to set up the experiment directory, lock the environment, rerun the baseline's published numbers as a calibration check, and seal the test set under a hash. ## Quick Start Set up a reproducible experiment directory with seeds, locked environment, and a calibration checkpoint before I run my first training job.

Frequently Asked Questions about reproducibility

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I make machine learning experiments reproducible?▼

Create the experiment directory before any run with config.yaml, seeds.txt, env.lock, run.sh, logs/, and outputs/. Record every run's command, seed, and git commit, and archive logs together with results so each number can be re-derived.

What is a calibration checkpoint in research pipelines?▼

A calibration checkpoint reproduces a known published result before running real experiments, validating the pipeline as an instrument. Only proceed to new experiments when the baseline's published numbers match in your environment.

How do I prevent test set leakage in ML experiments?▼

Split the test set before any preprocessing and physically isolate it from all training computation. Archive it under a hash with separate split and training scripts so training paths cannot reach it until final evaluation.

Why do my results change across machines or library versions?▼

Environment drift from unlocked dependencies changes results silently. Lock the environment with env.lock or a container, record versions, and rerun the calibration checkpoint after every environment change to catch drift immediately.

When is a README not enough for reproducibility?▼

A README written after results cannot capture which seed, environment, or preprocessing produced each number. Replay is the real test: the experiment directory must contain configs, per-run scripts, and logs so the README only describes how to replay.