skills.rest
beelabstudio
beelabstudioOFFICIAL
Explore all skills by this author →
GitHub

data-engineer

Designs data pipelines, ETL/ELT processes, and data warehouse architectures.

Updated Mar 14, 2026
Search Tags:#data pipeline#dbt#data engineering#etl#airflow#data warehouse#spark
One-click install
npx skills add https://github.com/beelabstudio/ai --skill data-engineer-beelabstudio
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-engineer
Source: https://github.com/beelabstudio/ai/tree/main/skills/foundation/data-engineer
Command: npx skills add https://github.com/beelabstudio/ai --skill data-engineer-beelabstudio

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Teams struggle to design scalable data infrastructure, choose the right ETL/ELT tooling, and maintain data quality across warehouses and lakes. This Skill provides a data engineering specialist persona that guides pipeline design, data modeling, and governance decisions. ## Core Features & Use Cases - Pipeline Design: Plan and implement batch and streaming data pipelines using Airflow, Prefect, Dagster, Spark, and Kafka. - Warehouse & Lake Architecture: Model data for Snowflake, BigQuery, or Redshift and define data lake structures. - Data Quality & Governance: Apply dbt tests, Great Expectations checks, and governance policies to keep data reliable. - Use Case: Ask for an end-to-end design of an ELT pipeline that ingests Kafka events into BigQuery, transforms them with dbt, and orchestrates the workflow with Airflow. ## Quick Start Ask the data engineer to design an ELT pipeline that loads daily sales data into Snowflake and models it with dbt.

Frequently Asked Questions about data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design an ETL pipeline with Airflow?▼

Define your pipeline as a DAG of tasks covering extraction, transformation, and loading stages. Airflow schedules and monitors execution, while tools like dbt or Spark handle the transformation logic inside individual tasks.

What is the difference between ETL and ELT processes?▼

ETL transforms data before loading it into the warehouse, while ELT loads raw data first and transforms it inside the warehouse using tools like dbt. ELT is common with cloud warehouses such as Snowflake and BigQuery.

Airflow vs Prefect vs Dagster for data orchestration?▼

Airflow is the established standard with a large ecosystem, Prefect offers dynamic workflows with simpler Python-native syntax, and Dagster emphasizes typed data assets and testing. Choose based on team familiarity and workflow complexity.

Can Kafka be used with Snowflake or BigQuery?▼

Yes, Kafka streams events that can be loaded into Snowflake via Snowpipe Streaming or Kafka Connect, and into BigQuery through Dataflow or connector sinks. This pattern supports near-real-time analytics on streaming data.

How do I ensure data quality in a data pipeline?▼

Add automated validation at each stage using dbt tests or Great Expectations to check schema, nulls, and value ranges. Combine these checks with monitoring dashboards and alerting so pipeline failures surface quickly.

Related Skills

Explore More Data & Analytics Skills→
Adelie-SquadAdelie-Squad
17

data-collector

Collect and aggregate data from APIs, web scraping, and event streams into a unified pipeline.

Official
Intermediate
2nth-ai2nth-ai
1

data-science

Guide machine learning model development from problem framing to evaluation.

Official
Advanced
BoraPerusicBoraPerusic

data-quality-frameworks

Validate data pipelines with Great Expectations, dbt tests, and data contracts.

Community
Advanced
sandeepmvlsandeepmvl
21

data-warehouse-integration

Design ELT pipelines syncing Rails Postgres to cloud warehouses with dbt.

Community
Advanced