add-incremental-loading

What problem does it solve? Filesystem pipelines that reload every file on each run waste time and compute, and replace-mode loads cannot deduplicate updated records. This Skill converts a working dlt filesystem pipeline so each run only reads new or modified files and merges records by primary key. ## Core Features & Use Cases - File-level incremental loading: Adds incremental=dlt.sources.incremental("modification_date") to the filesystem() call so only new or modified files are read. - Record-level filtering and deduplication: Applies apply_hints with a primary key and timestamp column, switching the write disposition from replace to merge (or append when no key exists). - State-safe configuration: Removes dev_mode=True so pipeline state persists across runs, then verifies behavior with a two-run test. - Use Case: You have a dlt pipeline loading CSVs from an S3 bucket in replace mode. Use this Skill to convert it so nightly runs only ingest files modified since the last run and upsert changed records by their ID column. ## Quick Start Convert my dlt filesystem pipeline to incremental loading with merge on the order_id primary key and updated_at as the record timestamp column.