Skills
.Work. You
Rest

Pipeline Design

Community

Design robust data pipelines.

Software Engineering#data quality#data pipeline#ETL#streaming#orchestration#batch processing#ELT
Authordtsong
Version1.0.0
Installs0

System Documentation

What problem does it solve?

This Skill helps design data pipelines that reliably move, transform, and deliver data from source systems to consumption layers, addressing key architectural decisions.

Core Features & Use Cases

  • ETL vs ELT Pattern Selection: Choose the most appropriate data loading strategy.
  • Batch vs Streaming: Decide between scheduled or continuous data processing.
  • Idempotency & Data Quality: Ensure pipelines are re-runnable and data integrity is maintained.
  • Lineage & Observability: Plan for tracking data flow and monitoring pipeline health.
  • Orchestration Tool Choice: Select the right tool (dbt, Airflow, Dagster, Prefect) based on team needs.
  • Use Case: Designing a new data ingestion process for customer transaction data, ensuring it meets freshness requirements, is auditable, and can be easily monitored.

Quick Start

Use the pipeline design skill to create a data flow diagram for ingesting user activity logs into a data warehouse.

Dependency Matrix

Required Modules

None required

Components

references

💻 Claude Code Installation

Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.

Please help me install this Skill:
Name: Pipeline Design
Download link: https://github.com/dtsong/claude-code-windows-setup/archive/main.zip#pipeline-design

Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
View Source Repository

Agent Skills Search Helper

Install a tiny helper to your Agent, search and equip skill from 223,000+ vetted skills library on demand.