silver-layer-setup

What problem does it solve?

This Skill automates the creation of robust, production-grade Silver layer pipelines using Spark Declarative Pipelines (SDP), ensuring data quality, efficient processing, and maintainability.

Core Features & Use Cases

• End-to-End Orchestration: Manages the setup of Silver layer pipelines from DQ rules table creation to DLT notebook generation and pipeline configuration.
• Data Quality Integration: Implements data quality checks and quarantine patterns directly within the pipeline using a centralized rules repository.
• Use Case: When establishing a new data product on Databricks, use this Skill to quickly scaffold the Bronze-to-Silver ETL process, incorporating best practices for data validation, streaming ingestion, and monitoring.

Quick Start

Use the silver-layer-setup skill to create the Silver layer for the transactions data.