skills.rest
shinnytech
shinnytechOFFICIAL
Explore all skills by this author →
WebsiteGitHub

senior-data-engineer

Design scalable data pipelines and ETL/ELT systems with Python, SQL, and Airflow.

Updated Jan 13, 2026
Search Tags:#scalability#data quality#ETL#orchestration#data engineering#data pipelines#dataops
One-click install
npx skills add https://github.com/shinnytech/caiwenqiang-member-rank --skill senior-data-engineer-shinnytech
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: senior-data-engineer
Source: https://github.com/shinnytech/caiwenqiang-member-rank/tree/main/.cursor/skills/senior-data-engineer
Command: npx skills add https://github.com/shinnytech/caiwenqiang-member-rank --skill senior-data-engineer-shinnytech

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the complexities of building and maintaining robust, production-grade data systems, enabling efficient data processing, reliable pipelines, and optimized data infrastructure.

Core Features & Use Cases

  • Data Pipeline Orchestration: Design, build, and manage complex data workflows using tools like Airflow.
  • Data Quality Validation: Implement automated checks to ensure data accuracy, completeness, and consistency.
  • ETL/ELT Optimization: Fine-tune data transformation processes for maximum performance and resource efficiency.
  • Use Case: A company needs to ingest data from multiple sources, transform it, and load it into a data warehouse daily. This Skill can orchestrate this entire process, validate the data at each stage, and optimize the transformations for speed.

Quick Start

Use the senior-data-engineer skill to orchestrate data pipelines starting from the 'data/' directory and outputting to 'results/'.

Frequently Asked Questions about senior-data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build scalable data pipelines with Airflow and dbt?▼

This Skill builds scalable data pipelines by orchestrating complex workflows with Airflow and dbt, managing data transformations and ensuring reliable execution from ingestion to warehouse loading.

What is DataOps and how does it improve ETL data quality?▼

DataOps improves ETL data quality by implementing automated validation checks throughout the pipeline, ensuring data accuracy, completeness, and consistency during transformation and loading processes.

Can I use this to optimize ETL workflows for Spark and Kafka?▼

Yes, you can optimize ETL workflows for Spark and Kafka by fine-tuning data transformation processes and resource efficiency, designing scalable infrastructure across the modern data stack.

What is the best way to orchestrate daily data warehouse loading?▼

The best way to orchestrate daily data warehouse loading is using Airflow for workflow management, ingesting from multiple sources, validating data at each stage, and optimizing transformations for speed.

How do I start orchestrating data pipelines from a local directory?▼

To start orchestrating data pipelines locally, configure the workflow to read input data from the 'data/' directory and output processed results to the 'results/' directory.

Related Skills

Explore More Software Engineering Skills→
AshishYesale7AshishYesale7
1

lg-kml-writer

Generate KML files with placemarks, tours, overlays, and polygons.

Community
Intermediate
faisalanjumfaisalanjum
3

earnings-orchestrator

Orchestrate multi-step earnings analysis pipelines for company tickers.

Community
Advanced
Fantasia1999Fantasia1999

senior-data-engineer

Design and implement scalable data pipelines using Python, SQL, Spark, Airflow, dbt, and Kafka.

Community
Advanced
faionfaionfaionfaion
4

faion-infrastructure-engineer

Automate cloud infrastructure provisioning across Docker, Kubernetes, Terraform, AWS, and GCP.

Community
Advanced