skills.rest
RajaPoseidon
RajaPoseidonCOMMUNITY
Explore all skills by this author →
GitHub

evaluate-cortex-analyst

Runs Snowflake Analyst Evaluations to measure Cortex Analyst SQL generation accuracy against verified queries.

Updated Jun 9, 2025
Search Tags:#evaluation#accuracy#snowflake#semantic view#sql correctness#cortex analyst#verified queries
One-click install
npx skills add https://github.com/RajaPoseidon/snippets_repo --skill evaluate-cortex-analyst-rajaposeidon
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluate-cortex-analyst
Source: https://github.com/RajaPoseidon/snippets_repo/tree/main/snowflake/skills/evaluate
Command: npx skills add https://github.com/RajaPoseidon/snippets_repo --skill evaluate-cortex-analyst-rajaposeidon

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? Measuring whether a Cortex Analyst semantic view generates correct SQL is hard to do systematically. This Skill runs Snowflake's native Analyst Evaluations against a semantic view's own verified queries, producing an accuracy score, surfacing regressions, and validating before/after improvements. ## Core Features & Use Cases - Verified-Query-Based Evaluation: Uses the semantic view's own verified queries as ground truth with the sql_correctness metric, with no separate dataset required. - Automated Run Management: Generates the YAML evaluation config, uploads it to a stage via PUT, starts the run with EXECUTE_AI_EVALUATION, and polls status until completion. - Results Analysis: Retrieves per-query results with GET_ANALYST_AI_EVALUATION_DATA, computes accuracy summaries, and inspects generated SQL versus verified SQL for incorrect rows. - Use Case: After refining a semantic view's instructions, re-run the evaluation to confirm the accuracy score improved and check that no previously correct verified queries regressed. ## Quick Start Ask the assistant to evaluate your Cortex Analyst semantic view by providing its database, schema, and view name, then follow the guided steps to review verified queries and run the evaluation.

Frequently Asked Questions about evaluate-cortex-analyst

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate a Cortex Analyst semantic view in Snowflake?▼

Use Snowflake's native Analyst Evaluations: generate a YAML config pointing to the fully qualified semantic view, upload it to a stage, and start a run with EXECUTE_AI_EVALUATION. The evaluation tests the view against its own verified queries using the sql_correctness metric.

What metrics are supported for Cortex Analyst evaluations?▼

Only sql_correctness is supported for Cortex Analyst evaluations. It executes the generated SQL and compares the result set against the verified query's result set. No dataset table or metric selection is involved.

Do Cortex Analyst evaluations require a separate dataset?▼

No. Unlike Cortex Agent evaluations, Analyst evaluations use the semantic view's own verified queries as the evaluation set. The view must have at least one verified query, added via Snowsight or the AI_VERIFIED_QUERIES clause.

Why does my Cortex Analyst evaluation run fail with permission errors?▼

Evaluation runs execute via tasks under a single primary role, so missing grants surface as confusing errors. Check for CORTEX_USER database role, EXECUTE TASK ON ACCOUNT, CREATE TASK and CREATE DATASET on the schema, SELECT and MONITOR on the semantic view, and SELECT on all referenced base tables.

Why do Cortex Analyst evaluation results change over time?▼

Verified queries referencing relative time concepts like 'last quarter' cause ground truth staleness, since verified and generated SQL return different rows on different days. Scope verified queries to absolute dates and time ranges for consistent results.

Related Skills

Explore More Data & Analytics Skills→
exploreomniexploreomni
30

omni-to-snowflake-semantic-view

Convert Omni Analytics topics into Snowflake Semantic View YAML definitions.

Official
Advanced
randoneeringrandoneering

evaluate-cortex-agent

Evaluate Cortex Agents in Snowflake and review results in Snowsight.

Community
Advanced
aaronachermannaaronachermann

databricks-metric-views

Define and manage YAML-based business metrics in Databricks Unity Catalog.

Community
Advanced
plurigridplurigrid
60

propagators

Propagate monotone values across interdependent cells until fixpoint convergence.

Official
Advanced