engineering-data-engineer

What problem does it solve?

This Skill addresses the complex challenges of designing, building, and operating robust data infrastructure, ensuring data is reliable, scalable, and accessible for analytics and AI.

Core Features & Use Cases

• Data Pipeline Engineering: Design and build ETL/ELT pipelines with idempotency, observability, and self-healing capabilities.
• Data Platform Architecture: Architect cloud-native data lakehouses and implement open table formats.
• Data Quality & Reliability: Define and enforce data contracts, monitor SLAs, and track data lineage.
• Streaming & Real-Time Data: Build event-driven pipelines using Kafka and stream processing frameworks.
• Use Case: A company needs to consolidate customer data from multiple sources into a unified, analytics-ready format. This Skill can design and implement the necessary Bronze, Silver, and Gold data pipelines using Spark and Delta Lake, ensuring data quality and timely delivery.

Quick Start

Use the engineering-data-engineer skill to design a Medallion Architecture data pipeline for customer order data.