What problem does it solve? Designing ETL/ELT pipelines across Databricks and Microsoft Fabric often leads to inconsistent layer patterns, misuse of streaming tables versus materialized views, and missing quality controls. This Skill enforces a standardized Medallion architecture (Bronze/Silver/Gold) with mandatory per-layer rules so pipelines are built correctly from the start. ## Core Features & Use Cases - Medallion Architecture Rules: Defines mandatory Lakeflow/SDP object types per layer, such as STREAMING TABLE with cloud_files() for Bronze and MATERIALIZED VIEW for Gold, including explicit prohibitions like manual SCD2 window functions. - Cross-Platform Patterns: Provides connectivity strategies between Fabric OneLake and Databricks via ABFSS paths, OneLake shortcuts, or export/upload workflows. - Databricks Job Configuration: Includes a JSON reference for multi-task jobs with dependencies, cron scheduling, and retry policies, plus a 10-point pipeline quality checklist. - Use Case: When asked to design a new sales data pipeline, apply this Skill to produce a Bronze ingestion layer via Auto Loader, a Silver layer using AUTO CDC INTO for SCD Type 2, and a Gold star schema, along with a scheduled Databricks job definition. ## Quick Start Use the pipeline-design skill to design a new ETL pipeline that ingests CSV files from a Fabric Lakehouse into Databricks following the Medallion architecture.
