ml-deployment

Deploy a machine learning model by first running it in shadow mode where its predictions are logged but discarded, then ramping a canary to 1-10% of traffic. Gate the canary on prediction distribution shift and business proxies, not error rates, since a broken model still returns healthy 200 responses.

Training/serving skew occurs when the features a model sees in production differ from those it was trained on, usually from reimplemented feature logic or re-fitted transformers. Prevent it by defining feature transformations once in a shared library and serializing the fitted preprocessor alongside the model.

Use shadow deployment first for every model since it catches schema mismatches and latency issues with zero live traffic. Follow with a canary for models touching revenue or users; blue-green only suits low-stakes or offline batch models because it detects infrastructure faults only.

Offline-online performance gaps usually come from point-in-time leakage, where training features were computed after the label event, or from serving-time feature divergence. Recompute training-side features for live entities in shadow mode and emit a mismatch metric when values diverge beyond tolerance.

Roll back a model by serving through an alias so promotion and rollback are pointer moves, such as an MLflow set-alias command, rather than image rebuilds. Keep the previous version warm in memory, since re-downloading a large artifact during an incident turns a rollback into an outage.