skills.rest
snoodleboot-io
snoodleboot-ioOFFICIAL
Explore all skills by this author →
GitHub

model-evaluation

Guides metric selection, honest data splitting, and uncertainty estimation for machine learning model evaluation.

Updated Dec 29, 2025
Search Tags:#calibration#model evaluation#cross-validation#data leakage#scikit-learn#classification metrics#roc-auc
One-click install
npx skills add https://github.com/snoodleboot-io/discrecontinual_equations --skill model-evaluation-snoodleboot-io
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: model-evaluation
Source: https://github.com/snoodleboot-io/discrecontinual_equations/tree/main/.claude/skills/model-evaluation
Command: npx skills add https://github.com/snoodleboot-io/discrecontinual_equations --skill model-evaluation-snoodleboot-io

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires scikit-learn, numpy.

What problem does it solve? Machine learning models often look strong offline but fail in production because of misleading metrics, leaky data splits, and overconfident single-number estimates. This Skill provides a rigorous evaluation methodology so reported results reflect real-world performance. ## Core Features & Use Cases - Metric Selection Guidance: Maps task types (imbalanced classification, regression, ranking) to appropriate metrics like PR-AUC, macro-F1, log loss, and NDCG, explaining why ROC-AUC flatters imbalanced data. - Leakage-Proof Splitting: Covers GroupKFold for repeated entities, TimeSeriesSplit for temporal data, and Pipeline-based preprocessing to prevent test-set contamination. - Uncertainty Quantification: Shows how to compute confidence intervals via repeated cross-validation and bootstrap, run paired model comparisons, tune decision thresholds from explicit costs, and check calibration. - Use Case: A data scientist evaluating a fraud detection model with 0.5% positive prevalence uses this Skill to report average precision instead of ROC-AUC, split by customer ID, and attach a bootstrap interval before claiming an improvement. ## Quick Start Ask the AI to evaluate your classifier honestly, choosing the right metric for imbalanced data and reporting confidence intervals with leakage-free cross-validation.

Frequently Asked Questions about model-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate a machine learning model on imbalanced data?▼

Use average precision (PR-AUC) instead of ROC-AUC when positives are rare. ROC-AUC hides false-positive volume because true negatives dominate the FPR denominator, while PR-AUC has no such cushion and reflects the precision your review team actually experiences.

What is the difference between ROC-AUC and PR-AUC?▼

ROC-AUC plots TPR against FPR and stays high on imbalanced data because true negatives inflate the FPR denominator. PR-AUC plots precision against recall with no true-negative term, so its baseline equals the positive rate and it honestly reflects rare-positive performance.

How do I prevent data leakage in cross-validation?▼

Fit all preprocessing inside a scikit-learn Pipeline so scalers and encoders refit within each fold. Use GroupKFold when the same entity appears in multiple rows and TimeSeriesSplit for temporal data so you never train on the future.

How do I choose a classification threshold?▼

Tune the threshold on validation data using the precision-recall curve or explicit error costs, then evaluate once on test. With known costs, the optimal threshold on calibrated probabilities is C_fp / (C_fp + C_fn), not the default 0.5.

Why is my model well-ranked but poorly calibrated?▼

Calibration and discrimination are independent: any monotone transform of scores leaves ROC-AUC unchanged while altering log loss. Fix calibration with CalibratedClassifierCV using isotonic regression for large datasets or sigmoid (Platt) scaling for small ones.

When should I not use accuracy as a metric?▼

Avoid accuracy on imbalanced data, since predicting the majority class at 1% prevalence yields 99% accuracy with zero value. Report average precision plus recall at a stated precision floor, and always compare against a DummyClassifier baseline.

Related Skills

Explore More Data & Analytics Skills→
chenghaoYangchenghaoYang
2

train

Run training experiments from recipe JSON files and store results.

Community
Advanced
KentoShimizuKentoShimizu
7

mlops-monitoring-drift

Detect data drift, concept drift, and quality degradation in production ML pipelines.

Community
Advanced
shalevaminshalevamin

cohort-analysis

Analyze user engagement data to reveal retention curves and feature adoption trends.

Community
Advanced
HolobiomicsLabHolobiomicsLab
4

pymc-bayesian-modeling

Infer posterior distributions and compare Bayesian models using PyMC.

Official
Advanced