dask

Use dask.dataframe to read the file lazily with dd.read_csv, which splits it into pandas-sized partitions processed in parallel. Operations build a task graph and only execute when you call .compute(), keeping memory usage bounded.

Switch to Dask when your dataset exceeds available RAM or pandas operations are too slow even after optimization. If data fits comfortably in memory and computations finish in seconds, plain pandas is simpler and faster.

Use Dask for scaling existing pandas/NumPy code beyond memory or across clusters. Use vaex for out-of-core analytics on a single machine, and polars for maximum in-memory speed on data that fits in RAM.

Yes, Dask reads remote Parquet and other formats from s3:// and gs:// paths. Install s3fs for S3 or gcsfs for Google Cloud Storage, and pass credentials via the storage_options parameter.

A slow start usually means the task graph is too large, often millions of tiny tasks from small chunks. Increase chunk sizes toward roughly 100 MB each or fuse operations with map_partitions or map_blocks.

Use threads (the default) for NumPy and pandas operations that release the GIL, processes for pure Python code, synchronous for debugging with pdb, and the distributed scheduler when you need the dashboard or multi-machine clusters.