dask

What problem does it solve?

Dask removes memory and single-core limits for pandas and NumPy workflows by enabling out-of-core execution, parallel processing, and distributed computation so users can analyze datasets that exceed a machine's RAM and accelerate slow data pipelines.

Core Features & Use Cases

• DataFrames: Parallelize pandas-style tabular workflows across partitions and multiple files for scalable ETL and aggregation.
• Arrays: Run large NumPy-style numerical computations on chunked arrays for scientific and image processing workloads.
• Bags: Process unstructured text, JSON, and logs in a streaming, memory-efficient fashion before converting to structured forms.
• Futures & Schedulers: Build dynamic, task-based parallel workflows with immediate execution, choose threads/processes/distributed schedulers, and monitor with a dashboard.
• Integration & IO: Native support for Parquet, Zarr, HDF5, and cloud-backed patterns for scalable storage and efficient serialization.
• Use Case Example: Run a multi-file ETL that reads many CSV/Parquet files, cleans and aggregates data using map_partitions, persists intermediates, and writes result sets to Parquet without exceeding memory.

Quick Start

Use Dask to read multiple CSV files as a single DataFrame, filter rows with value > 100, and compute the per-category mean.