transforming-data

Configure the model with materialized='incremental' and a unique_key, then add a conditional where clause using is_incremental() to filter records newer than the maximum timestamp in the existing table. This processes only new or changed rows on each run.

Use pandas for datasets under 500MB and prototyping; use polars for 500MB-100GB production pipelines where its multi-threaded lazy evaluation runs 10-100x faster. For datasets over 100GB requiring distributed clusters, choose PySpark.

Use ELT with a modern cloud warehouse like Snowflake or BigQuery when transformation logic changes frequently and data volume is large. Use ETL when compliance requires pre-load PII redaction or the target system lacks compute power.

Replace read_csv with scan_csv for lazy evaluation, convert boolean indexing to filter(pl.col(...)), replace assign with with_columns, and add collect() to execute the lazy query. Most pandas operations have direct polars equivalents.

Choose Airflow for enterprise production with thousands of integrations, Dagster for dbt-heavy workflows with asset-based lineage, and Prefect for dynamic workflows with a Pythonic decorator API. Airflow is the safe default for most teams.

Duplicates occur when the unique_key is missing or the incremental filter overlaps previously loaded data. Set a unique_key with the merge strategy and use a lookback window to handle late-arriving data safely.