data-pipeline-design

What problem does it solve?

This Skill addresses the complexities of designing, building, and maintaining reliable data pipelines, ensuring data integrity, efficient processing, and operational visibility.

Core Features & Use Cases

• Pipeline Pattern Selection: Guides users in choosing between ETL, ELT, and streaming patterns based on specific requirements.
• Stage Design: Provides detailed guidance on designing extract, transform, and load stages, including incremental loading, validation, and upsert strategies.
• Error Handling & Monitoring: Emphasizes robust error handling, dead-letter queues, retry mechanisms, alerting, and comprehensive monitoring for data quality and pipeline operations.
• Idempotency & Checkpointing: Ensures pipelines can be re-run safely and efficiently by implementing idempotency and checkpointing for long-running processes.
• Use Case: Design an ELT pipeline to ingest daily sales data from multiple APIs into a data warehouse, ensuring data is validated, transformed into a star schema, and monitored for any processing anomalies.

Quick Start

Use the data-pipeline-design skill to design an ETL pipeline for ingesting customer data from a PostgreSQL database into an S3 data lake.