data-eng
CommunityBuild reliable data pipelines.
Authorelihuvillaraus
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill automates the creation and maintenance of robust, scalable data pipelines and lakehouse architectures, transforming raw data into trusted, analytics-ready assets.
Core Features & Use Cases
- ETL/ELT Pipeline Development: Design and build idempotent, observable, and self-healing data pipelines.
- Lakehouse Architecture: Implement Medallion Architecture (Bronze, Silver, Gold) on cloud platforms.
- Data Quality & Reliability: Enforce data contracts, monitor SLAs, and implement lineage tracking.
- Streaming Data: Build event-driven pipelines with Kafka and stream processing frameworks.
- Use Case: Automatically ingest data from multiple sources, cleanse and conform it in the Silver layer, and aggregate it into business-ready metrics in the Gold layer, ensuring data quality and timely delivery.
Quick Start
Use the data-eng skill to build a bronze layer pipeline for ingesting JSON data from '/path/to/source' into 's3://my-bucket/bronze/events'.
Dependency Matrix
Required Modules
pysparkdbt-coregreat_expectationskafka-python
Components
scriptsreferences
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: data-eng Download link: https://github.com/elihuvillaraus/skills/archive/main.zip#data-eng Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 223,000+ vetted skills library on demand.