What problem does it solve? Predictive models often score brilliantly in validation yet fail on the units you actually act on, because of data leakage, random splits that ignore time or entity structure, and metrics that don't match the decision. This Skill enforces an honest evaluation discipline so a reported score survives deployment. ## Core Features & Use Cases - Prediction Spec workflow: Write a seven-row spec (decision, label regime, as-of prediction time, leakage audit, deployment-matched split, metric, baseline) and get sign-off before fitting anything. - Honest evaluation harness: Permutation/null probe, temporal or grouped splits, nested CV with structure-aware inner loops, calibration checks, and trivial-baseline comparisons. - Four label regimes: Guidance for clean labels, proxy/weak labels with selection bias, unsupervised anomaly detection, and ranking/triage at fixed capacity. - Use Case: When ranking pharmacies by diversion risk for a fixed audit capacity, the Skill forces a grouped temporal split, a permutation null check against the base rate, and precision-at-capacity reporting instead of a misleading 0.99 AUC. ## Quick Start Use the predictive-modeling skill to build and honestly evaluate a model that ranks claims by fraud risk for a weekly audit queue.
