mlflow-genai-evaluation

What problem does it solve?

This Skill streamlines the evaluation of Databricks GenAI agents by providing robust patterns for using MLflow's GenAI evaluation capabilities, ensuring agents meet quality and safety standards before deployment.

Core Features & Use Cases

• LLM-as-Judge Evaluation: Implement sophisticated evaluation pipelines using LLM judges for nuanced quality assessment.
• Custom Scorer Development: Create domain-specific evaluation metrics and integrate them seamlessly.
• Production Monitoring Setup: Configure Unity Catalog trace ingestion for continuous monitoring of agent performance in production.
• Use Case: Before deploying a customer support agent, use this Skill to run it against a benchmark dataset, evaluate its responses for accuracy, safety, and adherence to company guidelines, and automatically check if it meets predefined quality thresholds.

Quick Start

Use the mlflow-genai-evaluation skill to evaluate your agent's responses against a defined dataset and set of guidelines.