skills.rest
zhizhunbao
zhizhunbaoCOMMUNITY
Explore all skills by this author →
GitHub

senior-data-engineer

Design and build data pipelines, ETL workflows, and data quality frameworks with Airflow, dbt, Spark, and Kafka.

1|2|Updated Mar 5, 2026
Search Tags:#data quality#data pipeline#dbt#etl#airflow#kafka#spark
One-click install
npx skills add https://github.com/zhizhunbao/textbook-rag --skill senior-data-engineer-zhizhunbao
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: senior-data-engineer
Source: https://github.com/zhizhunbao/textbook-rag/tree/main/.agent/skills/dev-senior_data_engineer
Command: npx skills add https://github.com/zhizhunbao/textbook-rag --skill senior-data-engineer-zhizhunbao

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires great_expectations, and includes scripts (resource) and references (resource) components.

What problem does it solve? Building production data pipelines requires coordinating many tools—Airflow for orchestration, dbt for transformation, Spark for processing, Kafka for streaming—and getting architecture decisions wrong leads to slow queries, data quality issues, and costly rework. This Skill provides structured workflows, decision frameworks, and ready-to-use code patterns for the entire data engineering lifecycle. ## Core Features & Use Cases - Pipeline Construction: Step-by-step workflows for batch ETL (PostgreSQL → dbt → Snowflake) and real-time streaming (Kafka → Spark Structured Streaming → Delta Lake) with complete code examples. - Data Quality Frameworks: Great Expectations suites, dbt tests, data contracts, and quality monitoring dashboards, plus a validation script for schema checks, profiling, and anomaly detection. - Architecture Guidance: Decision frameworks for batch vs streaming, Lambda vs Kappa, warehouse vs lakehouse, plus dimensional modeling, SCD types, and Data Vault patterns in the reference docs. - Use Case: You need to build a daily incremental pipeline from PostgreSQL to Snowflake with data quality gates. Follow Workflow 1 to generate the Airflow DAG, dbt incremental models, and test suites, then validate output with the data quality validator script. ## Quick Start Ask the assistant to design a daily incremental ETL pipeline from PostgreSQL to Snowflake with dbt models and data quality tests.

Frequently Asked Questions about senior-data-engineer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build an incremental ETL pipeline with dbt and Airflow?▼

Create a dbt model with materialized='incremental' and a unique_key, filtering on a watermark column like updated_at when is_incremental() is true. Then orchestrate extract, dbt run, and dbt test tasks in an Airflow DAG with retries and alerting.

Should I use batch or streaming for my data pipeline?▼

Choose streaming when you need second-to-minute latency for continuous event data, using Kafka with Spark Structured Streaming or Flink. Choose batch with dbt and a warehouse when latency of hours is acceptable, since it is simpler and more cost-effective.

What is the difference between Lambda and Kappa architecture?▼

Lambda maintains separate batch and speed layers, suiting ML training on historical data plus real-time serving. Kappa uses a single stream processing path with replay from a long-retention Kafka log, reducing codebase duplication and maintenance.

How do I implement slowly changing dimensions in dbt?▼

Use an incremental model with strategy='check' and check_cols listing the tracked attributes, generating surrogate keys and effective date ranges. The reference documentation covers SCD Types 0 through 6 with full SQL implementations.

Why is my Spark job running out of memory?▼

Spark OOM errors typically come from undersized executors or skewed partitions. Increase spark.executor.memory, tune spark.sql.shuffle.partitions to match data volume, and enable adaptive query execution to coalesce partitions automatically.

How do I handle late-arriving data in Spark Structured Streaming?▼

Add a watermark with withWatermark('event_time', '10 minutes') so state is dropped after the threshold, and route records missing required fields to a dead letter queue using foreachBatch with custom error handling.

Related Skills

Explore More Data & Analytics Skills→
ivanshamaevivanshamaev
14

redpanda

Deploy and tune Redpanda clusters with rpk, Schema Registry, and tiered storage.

Community
Advanced
sandeepmvlsandeepmvl
21

data-warehouse-integration

Design ELT pipelines syncing Rails Postgres to cloud warehouses with dbt.

Community
Advanced
timequitytimequity
6

data-quality

Detect data quality issues in dbt and Great Expectations pipelines.

Official
Advanced
nrakownrakow
1

ingestion-strategy

Select and configure data ingestion strategies for Fivetran, Airbyte, or custom pipelines.

Community
Intermediate