ml-training-pipeline

What problem does it solve?

This Skill provides a concise, production-oriented blueprint to prevent silent failures and wasted compute during model training by enforcing correct data pipelines, distributed verification, mixed-precision, and rigorous checkpointing.

Core Features & Use Cases

• DDP-first verification: Start with single-node multi-GPU DDP verification (e.g., torchrun overfit-one-batch) to catch distributed bugs locally before scaling.
• Mixed precision and checkpointing: Enable bf16 mixed precision from day one and always save model, optimizer, scheduler, step, and RNG states with verified reloads to ensure training continuity.
• Scaling guidance and guardrails: Clear decision rules for when to adopt multi-node DDP or FSDP, plus a DDP silent-failure checklist covering logging, checkpoint rank, effective batch size, and seed handling.

Quick Start

Launch a two-GPU DDP training run with bf16 mixed precision, checkpointing every 500 steps, and verify overfit-one-batch succeeds.