What problem does it solve? Production ML models degrade silently: labels arrive weeks late, per-feature drift tests flood alert channels with false positives, and aggregate metrics hide failing segments. This Skill provides the patterns to build a monitoring stack that pages on real problems and distinguishes data bugs from genuine drift. ## Core Features & Use Cases - Layered Alerting Design: Orders monitoring layers by detection latency, from operational metrics to business outcomes, so fast layers page first. - Prediction Drift with PSI: Implements Population Stability Index as an effect-size drift metric that does not degrade at high sample volumes, with threshold-based alert rules. - Delayed Ground Truth Compensation: Covers lagged quality metrics, fast proxy labels, and randomized control holdouts to handle label maturity delays and selection bias. - Use Case: A fraud model's chargeback labels take 60 days to arrive. Use this Skill to set up lagged AUC dashboards, proxy labels from manual review outcomes, and a hash-bucketed control holdout for unbiased error estimates. ## Quick Start Ask the assistant to design a monitoring and alerting plan for a production churn model whose labels arrive 60 days after prediction.
