databricks-spark-declarative-pipelines

What problem does it solve?

This Skill automates the creation, configuration, and updating of Databricks Lakeflow Spark Declarative Pipelines (SDP/LDP), enabling efficient data engineering workflows.

Core Features & Use Cases

• Pipeline Orchestration: Create and manage serverless data pipelines on Databricks.
• Data Ingestion: Supports Auto Loader for efficient ingestion from cloud storage, Kafka, Event Hubs, and Kinesis.
• Data Transformation: Implements streaming tables, materialized views, and advanced patterns like SCD Type 1/2 and CDC.
• Use Case: Set up a new data pipeline to ingest streaming data from Kafka, clean and transform it into silver tables, and then aggregate it into gold tables for business reporting, all managed through a single, declarative configuration.

Quick Start

Use the databricks-spark-declarative-pipelines skill to initialize a new Python pipeline project named 'customer_orders_pipeline' in the current directory, targeting the 'main' catalog and using personal schemas for development.