dask

Use dask.dataframe's read_csv to load the file lazily in partitions, then apply pandas-like operations and call compute() only on the final result. Dask processes chunks in parallel without loading the entire dataset into memory.

Switch to Dask when your dataset exceeds available RAM or pandas computations take too long despite optimization. If data fits comfortably in memory and operations complete in seconds, pandas remains the better choice.

Use Dask for scaling existing pandas/NumPy code across clusters or for distributed ML. Use vaex for out-of-core analytics on a single machine, and polars for maximum in-memory speed on data that fits in RAM.

Use threads (the default) for NumPy and pandas operations that release the GIL, processes for pure Python code, synchronous for debugging with pdb, and the distributed scheduler when you need the monitoring dashboard or multi-machine clusters.

Memory errors usually come from chunks that are too large or too many persisted intermediate results. Decrease chunk sizes to roughly 100 MB each, persist strategically, and delete cached objects when finished.

A slow start typically means the task graph is too large, often millions of tiny tasks. Increase chunk sizes, fuse operations with map_partitions or map_blocks, and check graph size with len(ddf.__dask_graph__()).