What problem does it solve? Building batch and streaming data pipelines on Databricks requires choosing the right dataset types, APIs, and deployment workflows, and mistakes like mismatched streaming/batch pairs or legacy DLT syntax cause validation errors and failed updates. ## Core Features & Use Cases - Decision Guidance: A decision tree maps user requirements to the correct dataset type (Streaming Table, Materialized View, Temporary View, Sink) and features like Auto Loader, Auto CDC, Append Flows, and Expectations. - API Reference Tables: Feature-to-API mappings for Python and SQL with links to detailed reference files covering CDC, sinks, Kafka ingestion, expectations, and table features like liquid clustering. - Workflow Scaffolding: Three project workflows covering standalone DAB bundles, pipelines in existing bundles, and rapid CLI iteration, including deploy, run, polling, and failure-diagnosis commands. - Legacy Migration: A complete mapping from legacy DLT syntax (import dlt, apply_changes, LIVE. prefix) to the modern pyspark.pipelines API. - Use Case: A data engineer asked to build a medallion pipeline ingesting JSON files from cloud storage into Bronze, cleansing into Silver with expectations, and aggregating into Gold gets scaffolded project files, correct streaming table definitions, and deploy commands. ## Quick Start Ask the agent to create a Databricks declarative pipeline that ingests files from a cloud storage path into a streaming table and aggregates the results into a materialized view.
