skills.rest
mnazaal
mnazaalCOMMUNITY
Explore all skills by this author →
Website𝕏@mnazaaIGitHub

dev-ml-infra

Configure reproducible ML experiment infrastructure with hydra-zen configs, tracker seams, and smoke tests.

Updated Jul 3, 2026
Search Tags:#reproducibility#experiment tracking#data leakage#smoke test#mlflow#hydra-zen#train test split
One-click install
npx skills add https://github.com/mnazaal/dotfiles --skill dev-ml-infra-mnazaal
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: dev-ml-infra
Source: https://github.com/mnazaal/dotfiles/tree/main/.agents/skills/dev-ml-infra
Command: npx skills add https://github.com/mnazaal/dotfiles --skill dev-ml-infra-mnazaal

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? ML research codebases often suffer from data leakage, non-reproducible runs, scattered configs, and silent result overwrites. This Skill provides a standard layout and set of conventions for building experiment infrastructure where every run is reproducible, comparable, and safe to launch concurrently. ## Core Features & Use Cases - Standard Research Layout: Defines a canonical repo structure with src/ library code, a single results/ output root, and one config-driven entry point (python -m project_name.run method=X data=Y seed=0). - Config-as-Code with hydra-zen: Replaces YAML config trees with typed builds()/make_config() definitions, CLI overrides, and a shared zen(main).hydra_main(...) launcher helper. - Tracker Seam: Routes all experiment logging through a small Tracker protocol so MLflow (default) or wandb can be swapped by changing one adapter module. - Leakage & Concurrency Safety: Enforces train-only preprocessing fits, grouped/temporal splits, seeded splits persisted as ID lists, atomic cache writes, and per-task output paths to avoid write races. - Use Case: When starting a new ML research repo, use this Skill to scaffold the layout, wire up hydra-zen configs and an MLflow tracker seam, and add an end-to-end smoke test that proves the full pipeline converges on tiny synthetic data. ## Quick Start Set up a reproducible ML experiment repo with hydra-zen configs, an MLflow tracker seam, and a smoke test using the dev-ml-infra skill.

Frequently Asked Questions about dev-ml-infra

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I prevent data leakage in ML train/test splits?▼

Fit preprocessing statistics on the training split only, since fit-then-split is the most common silent leak. Split on the grouping unit rather than the row, persist seeded splits as ID lists, and include the preprocessing code version in cache keys.

How do I manage ML experiment configs with hydra-zen instead of YAML?▼

Define configs as code using builds(), just(), and make_config() next to the components they configure, with typed fields and smoke-safe defaults. Expose the entry point via zen(main).hydra_main(...) so parameters become key=value CLI overrides and lists become xs=[a,b,c].

MLflow vs wandb for experiment tracking, how do I switch?▼

Depend experiment code on a small Tracker protocol with start_run, log_params, log_metrics, log_artifact, and end_run methods. Keep import mlflow in exactly one adapter module so swapping to wandb requires changing only that single file.

Why do progress bars break in SLURM or nohup logs?▼

tqdm carriage-return bars explode into thousands of junk lines when output is captured rather than attached to a TTY. Switch on sys.stderr.isatty() to emit periodic structured lines like step/loss/lr/eta instead, which also serve as a greppable record.

How do I stop concurrent ML runs from overwriting each other's results?▼

Make each run's output path unique by its full sweep key, not a partial key plus a wall-clock timestamp. Give each array task its own output file or directory and merge at read time, and write caches and checkpoints to a temp path then rename atomically.

What should an ML training pipeline smoke test cover?▼

Run the full pipeline through the real entry point on tiny synthetic data with a known answer for about 50 steps. Assert that loss decreases, metrics and artifacts land in the tracker, and the checkpoint reloads, without mocking the tracker or bypassing the entry point.

Related Skills

Explore More Education & Research Skills→
AratKruglikAratKruglik
9

python-app-conventions

Organize Python projects with src/ layout, CLI entry points, and pyproject.toml packaging.

Community
Intermediate
dogekikidogekiki

analyze-results

Parse JSON and CSV experiment files to compute performance statistics and comparative metrics.

Community
Intermediate
matsunagalabmatsunagalab
11

HPC Run

Submit and manage SLURM-based MDClaw workloads on HPC clusters.

Official
Advanced
p-o-ke-naep-o-ke-nae

scripting-guide

Run C# file-based data processing scripts via dotnet run in PowerShell or Bash.

Community
Intermediate