python-data-engineering

What problem does it solve?

This Skill provides practical patterns and templates to build robust data pipelines in Python using SQLAlchemy 2.0, enabling clean separation between source systems, the data warehouse, and consuming applications. It covers advanced modeling patterns (TypeDecorator, hybrid properties, events), dimensional modeling conventions (dim_/fact_/stg_), slowly changing dimensions, and scalable ETL/ELT architectures with modern orchestration tools.

Core Features & Use Cases

• Factory-based data transformation: convert API responses or raw data into strongly typed SQLAlchemy models with safe upserts and merges.
• Dimensional modeling and warehouse patterns: apply Kimball-style star schemas, medallion layers, and SCD strategies to production analytics.
• Incremental and resilient pipelines: implement high-water-mark incremental sync, JSONB-based schema resilience, and easy integration with dbt, Airflow, or Dagster.
• End-to-end templates for multi-app reuse: single source of truth for transformations used across multiple data apps and services.

Quick Start

Create a minimal data pipeline by defining a SQLAlchemy model, a from_api_response factory, and a small ETL that reads JSON data and writes to PostgreSQL.