skills.rest
ThomazRossito
ThomazRossitoCOMMUNITY
Explore all skills by this author →
Website𝕏@thomazrossitoGitHub

databricks-observability-migration

Maps SQL Server monitoring to Databricks system tables, alerts, and tuning workflows.

4|1|Updated May 22, 2026
Search Tags:#observability#databricks#system tables#lakeflow#sql alerts#sql server migration#delta optimization
One-click install
npx skills add https://github.com/ThomazRossito/ai-data-agents --skill databricks-observability-migration-thomazrossito
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-observability-migration
Source: https://github.com/ThomazRossito/ai-data-agents/tree/main/plugins/ai-data-agents/skills/databricks-observability-migration
Command: npx skills add https://github.com/ThomazRossito/ai-data-agents --skill databricks-observability-migration-thomazrossito

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? After migrating from SQL Server to Databricks, operations teams lose familiar monitoring tools like Query Store, SQL Agent job history, and Resource Governor, leaving them without equivalent visibility into query performance, job execution, and resource consumption. ## Core Features & Use Cases - Monitoring Equivalence Mapping: Maps legacy SQL Server DMVs and tools (Query Store, sysjobhistory, Resource Governor) to Databricks system tables such as system.query.history, system.lakeflow.job_run_timeline, and system.access.audit. - Lakeflow Pipeline Observability: Compares update-level monitoring via system tables against flow-level detail via the event_log TVF or configured event log tables. - Alerts as Code: Defines SQL alerts as versioned resources.alerts in Databricks Asset Bundles with thresholds, cron schedules, and notification destinations (Email, Slack, Teams, PagerDuty). - Performance Tuning Sequence: Applies CLUSTER BY, OPTIMIZE, and ANALYZE with Predictive Optimization for automated Delta table maintenance. - Use Case: After cutover, an operations engineer needs to detect failed migration jobs hourly and compare job success rates against the pre-migration msdb baseline; this Skill provides the exact SQL queries and DAB alert YAML to do so. ## Quick Start Ask the AI to set up post-migration monitoring for Databricks jobs using system tables and create a failure alert as code with Databricks Asset Bundles.

Frequently Asked Questions about databricks-observability-migration

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I monitor Databricks jobs after migrating from SQL Server?▼

Query system.lakeflow.job_run_timeline joined with system.lakeflow.jobs to get run results, durations, and termination codes. Compare these metrics against the msdb.dbo.sysjobhistory baseline captured before cutover to validate SLAs.

What replaces SQL Server Query Store in Databricks?▼

The system.query.history table replaces Query Store and sys.dm_exec_query_stats for query performance analysis and slow query detection. It retains 30 days of history, so export data to the data lake if longer retention is needed.

System tables vs event_log for Lakeflow pipeline monitoring?▼

System tables like system.lakeflow.pipeline_update_timeline provide update-level metrics across the account with no setup. The event_log TVF or a configured event log table provides flow-level detail such as rows written and expectations for a specific pipeline.

How do I create Databricks SQL alerts as code?▼

Define resources.alerts in the databricks.yml of a Databricks Asset Bundle with query_text, evaluation threshold, quartz cron schedule, and notification subscriptions. This makes alerts versioned and reproducible across dev, staging, and production environments.

Why is my Databricks alert not firing?▼

Check that the warehouse_id is valid, the quartz_cron_schedule syntax is correct, and empty_result_state is set appropriately. An alert with empty_result_state set to OK stays silent when the query returns no rows.

Does Predictive Optimization work on all Delta tables?▼

Predictive Optimization only runs on Unity Catalog managed tables, not external tables or Delta Sharing tables. Verify enablement at the schema, catalog, or account level using DESCRIBE SCHEMA EXTENDED.

Related Skills

Explore More Data & Analytics Skills→
motherduckdbmotherduckdb
53

motherduck-migrate-to-motherduck

Create phased, rollback-aware migration plans to MotherDuck from Snowflake, Redshift, and Postgres.

Official
Advanced
altrupetsaltrupets

Alerting & Incident Management

Automate alerting and incident management setup with Prometheus, Alertmanager, and PagerDuty.

Official
Advanced
phoenixvcphoenixvc

cost-centres

Manage cost centre configurations for cloud infrastructure budgets and tag compliance.

Official
Intermediate
databrickslabsdatabrickslabs
38

databricks-asset-bundles

Configure Databricks Asset Bundles for multi-environment deployments.

Official
Intermediate