Model Evaluation Patterns

What problem does it solve?

This Skill addresses the critical need to thoroughly evaluate machine learning models beyond simple accuracy, ensuring they are robust, fair, and performant across various scenarios.

Core Features & Use Cases

• Comprehensive Metrics: Generates a full suite of classification metrics with confidence intervals.
• Calibration Analysis: Assesses if model probabilities accurately reflect true likelihoods.
• Sliced Evaluation: Identifies performance disparities across different data subgroups.
• Fairness Metrics: Quantifies bias related to sensitive attributes.
• LLM Evaluation: Integrates BERTScore and RAGAS for generative model assessment.
• Use Case: After training a churn prediction model, use this Skill to generate a complete evaluation report, check its calibration, and verify it doesn't unfairly penalize specific customer segments.

Quick Start

Generate a full evaluation report for the test data using the provided true labels, predicted labels, and predicted probabilities.