senior-data-engineer

What problem does it solve?

This Skill provides senior-level guidance and tooling patterns for building scalable data pipelines, ETL/ELT architectures, governance, and DataOps in production-grade environments. It helps teams design robust data infrastructure and processes from planning to deployment.

Core Features & Use Cases

• End-to-end data pipeline design: ingestion, transformation, orchestration, and monitoring for large-scale datasets.
• Data governance and quality patterns: validation, lineage, security, and compliance in data workflows.
• Production ops guidance: deployment patterns, observability, cost optimization, and team leadership guidance.

Quick Start

Design a production-grade data pipeline for ingesting 1TB of daily logs, orchestrated with Airflow, processed with Spark, validated with data quality checks, and monitored with Prometheus or similar observability tools.