data-engineering-data-pipeline

What problem does it solve?

This Skill addresses the complexity of designing, implementing, and managing scalable, reliable, and cost-effective data pipelines for both batch and streaming data.

Core Features & Use Cases

• Architecture Design: Create ETL/ELT, Lambda, Kappa, and Lakehouse architectures.
• Implementation: Build ingestion, transformation (dbt, Spark), and orchestration (Airflow, Prefect) layers.
• Data Quality & Storage: Implement data quality frameworks and manage data in Delta Lake/Iceberg.
• Monitoring & Optimization: Ensure pipeline health and optimize costs.
• Use Case: Design a real-time streaming data pipeline to ingest user clickstream data, process it for sessionization, and store it in a Lakehouse for analytics.

Quick Start

Use the data-engineering-data-pipeline skill to design a batch ETL architecture for processing daily sales transactions from a PostgreSQL database into a Delta Lake table.