optimize-filesystem-performance

First measure extract time with pipeline.last_trace to confirm reading files is the bottleneck. Then apply matching levers: use a faster reader like read_csv_duckdb, stream in chunks, parallelize reads across pages, or narrow the file_glob to scan fewer objects.

Stream the file in chunks instead of loading it whole. Use read_csv with a chunksize parameter, read_csv_duckdb with chunk_size, or write a custom dlt transformer that yields pandas chunks via pd.read_csv(f.open(), chunksize=...).

read_csv_duckdb is faster and lower-memory than the pandas-based read_csv, and supports use_pyarrow=True for Arrow output. Overall read speed by format is Parquet fastest, then JSONL, then CSV.

Yes, call .parallelize() on the resource, but it splits reads across pages so files_per_page must be set well below your file count. Parallelization only helps remote object-store reads; local files are CPU and GIL bound.

The pipeline lacks file-level incremental loading, so it scans and reads every object each run. Set up incremental filtering by modification_date first, since reading fewer files beats reading them faster.

Use fsspec directly when you only need to move or copy files rather than parse their contents. The fsspec_from_resource helper provides an authenticated client for bulk operations, avoiding parse overhead and memory use entirely.