genie-benchmark-generator
OfficialGenerate & validate Genie Space benchmarks.
Authordatabricks-solutions
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill automates the creation, validation, and synchronization of benchmark questions for Genie Space optimization, ensuring robust evaluation of AI agent performance.
Core Features & Use Cases
- Multi-path Intake: Handles user-provided questions (10+, 1-9), or generates synthetic benchmarks from scratch.
- Ground Truth Validation: Executes SQL queries against the live warehouse to verify correctness and store results.
- MLflow Integration: Syncs validated benchmarks to MLflow Evaluation Datasets for seamless integration with GenAI evaluation workflows.
- Use Case: Before optimizing a Genie Space for cost analysis, use this Skill to generate a comprehensive set of benchmark questions, validate their expected SQL, and prepare them for MLflow evaluation, ensuring the optimization loop has reliable test cases.
Quick Start
Use the genie-benchmark-generator skill to create and validate benchmarks for the 'cost' domain in the 'main.genie_benchmarks' Unity Catalog schema.
Dependency Matrix
Required Modules
None requiredComponents
scriptsreferencesassets
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: genie-benchmark-generator Download link: https://github.com/databricks-solutions/vibe-coding-workshop-template/archive/main.zip#genie-benchmark-generator Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 223,000+ vetted skills library on demand.