ml-evaluation-framework

What problem does it solve?

This skill enforces rigorous evaluation practices to prevent unsupported claims about model performance by requiring variance estimation, confidence intervals, and fair comparisons before reporting results.

Core Features & Use Cases

• Statistical Rigor Enforcement: Requires minimum multiple random seeds, mean ± std reporting, and confidence intervals for any claimed improvement.
• Evaluation Checklist: Mandates user-specified metrics, full classification metric suite when applicable, ablations for novel components, and fair baseline comparisons using identical data splits and preprocessing.
• Use Case: Use when preparing benchmark reports, publishing model improvements, running ablation studies, or comparing models to ensure conclusions are statistically justified.

Quick Start

Ask the assistant to design an evaluation plan that lists metrics, specifies at least three seeds, defines baselines with identical splits, and outlines ablation experiments.