skills.rest
bhaktofmahakal
bhaktofmahakalCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

senior-data-engineer

Designs data pipelines, ETL workflows, and data infrastructure using Python, SQL, Spark, and Airflow.

2|1|Updated Jan 29, 2026
Search Tags:#data quality#data pipeline#data engineering#etl#airflow#dataops#spark
One-click install
npx skills add https://github.com/bhaktofmahakal/ai-counsellor-hf --skill senior-data-engineer-bhaktofmahakal
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: senior-data-engineer
Source: https://github.com/bhaktofmahakal/ai-counsellor-hf/tree/main/.claude/skills/senior-data-engineer
Command: npx skills add https://github.com/bhaktofmahakal/ai-counsellor-hf --skill senior-data-engineer-bhaktofmahakal

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve? Building scalable data pipelines and ETL/ELT systems requires deep expertise across orchestration, data modeling, quality validation, and DataOps, which is hard to apply consistently without structured guidance. ## Core Features & Use Cases - Pipeline Orchestration: Provides a pipeline_orchestrator.py script and architecture references for designing and running distributed data workflows. - Data Quality Validation: Includes a data_quality_validator.py tool to check datasets against configurable rules before they enter production pipelines. - ETL Performance Optimization: Offers an etl_performance_optimizer.py script plus references on data modeling patterns and DataOps best practices. - Use Case: When designing a new analytics platform, use this Skill to plan the pipeline architecture, validate incoming data quality, and optimize ETL job performance with Spark, Airflow, and dbt patterns. ## Quick Start Ask the AI to design a scalable ETL pipeline architecture for your data sources and validate the data quality of your target dataset.

Frequently Asked Questions about senior-data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design a scalable data pipeline architecture?▼

Start with the data pipeline architecture reference, which covers distributed processing, fault-tolerant design, and real-time versus batch patterns. Then use the pipeline_orchestrator.py script to coordinate workflow execution across your data sources.

How to validate data quality before loading into a warehouse?▼

Run the data_quality_validator.py script with an input path and output path to check datasets against configurable rules. It returns a JSON result with processing status, making it easy to integrate into CI or pipeline stages.

What tools does this skill cover for data engineering?▼

The skill covers Python, SQL, Spark, Airflow, dbt, Kafka, and Databricks for data workflows, plus Docker, Kubernetes, and AWS/GCP/Azure for deployment. It also references PostgreSQL, BigQuery, and Snowflake as target databases.

Can I use this for real-time streaming data pipelines?▼

Yes, the references cover real-time systems with low-latency, high-throughput design patterns, and the tech stack includes Kafka for streaming ingestion. Batch and real-time processing patterns are both documented.

What are the limitations of the included ETL scripts?▼

The scripts provide production-style scaffolding with config validation and structured logging, but the core execution logic is a template you must implement for your specific data sources. They are starting points, not turnkey solutions.

Related Skills

Explore More Data & Analytics Skills→
MapleTechLabsMapleTechLabs
1.6k

chdb-sql

Execute ClickHouse SQL queries on local files and remote databases in Python.

Official
Intermediate
finsilabsfinsilabs
44

ecommerce-data-warehouse

Build e-commerce data warehouses with star-schema, ETL, and dbt.

Official
Advanced
timequitytimequity
6

data-governance

Enforce data governance across data pipelines with lineage tracking and access controls.

Official
Advanced
v1bh0rv1bh0r

nw-data-architecture-patterns

Identify and apply data architecture patterns for data platforms.

Community
Intermediate