skills.rest
ice-droid-99
ice-droid-99COMMUNITY
Explore all skills by this author →
GitHub

agent-evaluation

Evaluate and improve LLM agent outputs using MLflow GenAI workflows.

Updated May 20, 2026
Search Tags:#llm evaluation#cost optimization#scorers#mlflow#dataset generation#agent tracing
One-click install
npx skills add https://github.com/ice-droid-99/BI-Dashboard-Generation-Agent-in-Databricks- --skill agent-evaluation-ice-droid-99
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agent-evaluation
Source: https://github.com/ice-droid-99/BI-Dashboard-Generation-Agent-in-Databricks-/tree/main/.gemini/skills/agent-evaluation
Command: npx skills add https://github.com/ice-droid-99/BI-Dashboard-Generation-Agent-in-Databricks- --skill agent-evaluation-ice-droid-99

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

It solves the problem of unreliable or expensive LLM agent outputs by providing a repeatable way to evaluate, diagnose, and improve agent behavior using MLflow’s native GenAI evaluation tooling.

Core Features & Use Cases

  • Quality evaluation end-to-end with MLflow: Runs the full evaluation workflow (tracing setup, dataset prep, scorer/judge setup, evaluation execution) using MLflow’s GenAI APIs.
  • Systematic scorer and dataset management: Discovers existing datasets first, registers scorers properly (including built-in scorers and custom judges), and uses MLflow-managed records instead of ad-hoc frameworks.
  • Operational guardrails for reliability and cost: Enforces tracing-first requirements, includes a required 3-question dry run, and provides runtime estimation and throughput guidance for larger evals.

Use case example: After an agent generates incorrect or incomplete answers for a BI/dashboard workflow, run this skill to trace failures, select/verify relevant scorers, evaluate on a generated 50–100 question dataset, and identify which criteria to fix.

Quick Start

Ask the AI to evaluate an existing agent by running an MLflow GenAI eval: include your agent’s purpose, choose or create an evaluation dataset, register scorers, run a 3-question dry run, then execute the full evaluation.

Frequently Asked Questions about agent-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM agent quality using MLflow?▼

MLflow GenAI evaluation assesses LLM agent quality by validating traces, preparing datasets, registering scorers or custom judges, and executing mlflow.genai.evaluate. It provides a repeatable workflow to diagnose and improve unreliable or expensive agent behavior.

What is the best way to set up MLflow tracing for agent evaluation?▼

Setting up MLflow tracing for agent evaluation requires using a dedicated tracing skill first to instrument and capture agent behavior. This tracing-first setup is mandatory to validate execution flows before registering scorers and running evaluations.

Do I need an existing dataset to run an MLflow GenAI evaluation?▼

You do not strictly need an existing dataset to run an MLflow GenAI evaluation. The workflow discovers existing datasets first, but also supports dataset generation to create a 50 to 100 question set for testing agent responses.

Can I use custom judges instead of built-in scorers for LLM evaluation?▼

You can use custom judges alongside built-in scorers for LLM evaluation. The MLflow workflow systematically registers scorers, allowing you to select and verify relevant custom judges or built-in options to measure specific agent criteria.

How do I estimate runtime and throughput for large LLM evaluations?▼

You can estimate runtime and throughput for large LLM evaluations using the workflow's operational guardrails. It enforces a required 3-question dry run and provides runtime estimation and throughput guidance to ensure reliability and cost optimization.

Why should I evaluate LLM agents component-by-component instead of end-to-end?▼

Evaluating LLM agents component-by-component isolates specific failure points within the workflow. MLflow GenAI evaluation supports both end-to-end and component-by-component modes, allowing you to diagnose and fix individual criteria rather than just overall behavior.

Related Skills

Explore More Software Engineering Skills→
JingyiWngJingyiWng

agent-evaluation

Orchestrate end-to-end GenAI agent evaluation with MLflow datasets, scorers, and tracing.

Community
Advanced
teegin-gteegin-g

agent-evaluation

Evaluate and optimize LLM agent outputs using MLflow APIs.

Community
Advanced
BlackkadderBlackkadder

databricks-mlflow-evaluation

Run MLflow GenAI evaluations with custom scorers, judges, and trace datasets.

Community
Advanced
m19cm19c

agent-evaluation

Evaluate LLM agent output quality using MLflow native APIs.

Community
Advanced