skills.rest
ThomazRossito
ThomazRossitoCOMMUNITY
Explore all skills by this author →
Website𝕏@thomazrossitoGitHub

databricks-mlflow-evaluation

Evaluate GenAI agents on Databricks using MLflow 3 scorers, traces, and judge alignment.

4|1|Updated May 22, 2026
Search Tags:#prompt optimization#tracing#MLflow#databricks#Unity Catalog#LLM judge#genai evaluation
One-click install
npx skills add https://github.com/ThomazRossito/ai-data-agents --skill databricks-mlflow-evaluation-thomazrossito
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-mlflow-evaluation
Source: https://github.com/ThomazRossito/ai-data-agents/tree/main/plugins/ai-data-agents/skills/databricks-mlflow-evaluation
Command: npx skills add https://github.com/ThomazRossito/ai-data-agents --skill databricks-mlflow-evaluation-thomazrossito

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires mlflow[databricks], openai, databricks-connect, and includes references (resource) components.

What problem does it solve? Evaluating LLM agents in production requires correct MLflow 3 GenAI APIs, Databricks-specific trace storage, and judge alignment with domain experts, and small API mistakes cause silent failures or misleading scores. ## Core Features & Use Cases - GenAI Evaluation Workflows: Write mlflow.genai.evaluate() code with built-in scorers (Guidelines, Correctness, Safety, RetrievalGroundedness) and custom @scorer functions, with 15+ documented gotchas preventing common API errors. - Unity Catalog Trace Ingestion & Monitoring: Store production traces in UC tables, configure trace destinations, and run continuous production monitoring with registered scorers and sampling rates. - Judge Alignment & Prompt Optimization: Align LLM judges with domain expert feedback using MemAlign labeling sessions, then run optimize_prompts() with GEPA for automated prompt improvement. - Use Case: A team deploys a RAG agent on Databricks and needs to build an evaluation dataset from production traces, align a quality judge with SME feedback, and automatically optimize the system prompt against that aligned standard. ## Quick Start Ask the AI to write an MLflow 3 GenAI evaluation for your agent using mlflow.genai.evaluate() with Safety and Guidelines scorers on a small dataset of test queries.

Frequently Asked Questions about databricks-mlflow-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate a GenAI agent with MLflow 3?▼

Use mlflow.genai.evaluate() with a dataset of records containing an inputs key, a predict_fn that receives unpacked kwargs, and a list of scorers such as Safety or Guidelines. Do not use the older mlflow.evaluate() API, which targets classic ML models.

How to create a custom scorer in MLflow GenAI?▼

Apply the @scorer decorator from mlflow.genai.scorers to a function accepting inputs, outputs, expectations, or trace, and return a boolean, float, string, or Feedback object. For production monitoring, place imports inside the function so the scorer serializes correctly.

What MLflow version is required for Unity Catalog trace ingestion?▼

Unity Catalog trace ingestion requires mlflow[databricks]>=3.9.0. You must also set MLFLOW_TRACING_SQL_WAREHOUSE_ID before linking a UC schema to an experiment, and grant explicit MODIFY and SELECT on the trace tables since ALL_PRIVILEGES is insufficient.

Why does MLflow judge alignment with MemAlign fail?▼

Alignment fails when the label schema name in the SME labeling session does not exactly match the judge name used in evaluate(), because align() pairs scores by name. Also note that episodic memory loads lazily, so inspect the judge's instructions field rather than _episodic_memory after get_scorer().

Why does optimize_prompts() with GEPA produce poor results?▼

GEPA requires every training record to include both inputs and expectations, unlike standard evaluation datasets. Without expectations, the optimizer cannot reason about why the current prompt underperforms, which is the most common cause of weak optimization outcomes.

Should I evaluate against a Databricks model serving endpoint during development?▼

No, import your agent module directly and call it in predict_fn for development. Local testing gives faster iteration, full stack traces, and no serving costs; reserve endpoints for production monitoring, load testing, or A/B testing deployed versions.

Related Skills

Explore More Data & Analytics Skills→
RamVegirajuRamVegiraju
4

agent-evaluation

Evaluate GenAI agents with MLflow datasets, scorers, and traces.

Community
Advanced
juanlamadrid20juanlamadrid20

agent-mlops

Automate production deployment and monitoring of AI agents on Databricks.

Community
Advanced
databricksdatabricks
183

add-tools

Configure Databricks agent access to external resources in databricks.yml.

Official
Intermediate
databrickslabsdatabrickslabs
38

databricks-mlflow-evaluation

Evaluate GenAI agents with MLflow 3 GenAI evaluation APIs and built-in scorers.

Official
Advanced