ray-data

Install ray[data] with pip, then use ray.data.read_parquet or read_csv to load data and apply map_batches for vectorized transformations. Ray Data streams execution, so datasets larger than memory are processed in batches without full loading.

Use Pandas for small data under 1GB on a single machine, Spark for enterprise SQL-based ETL, and Ray Data for ML workloads over 100GB needing distributed preprocessing, GPU acceleration, and direct integration with training frameworks.

Yes, Ray Data supports GPU transforms by passing num_gpus=1 to map_batches. The documentation shows image preprocessing scaling from 1,000 images per second on CPU to 18,000 per second with four GPUs.

Yes, convert datasets with ds.to_torch or ds.to_tf to feed batches directly into training loops. With Ray Train, use ray.train.get_dataset_shard to automatically shard datasets across training workers.

Ray Data reads and writes Parquet, CSV, JSON, and NumPy, and reads images and Pandas DataFrames. Parquet is the recommended format for ML data due to efficient columnar storage and streaming reads.

Avoid Ray Data for small datasets under 1GB where Pandas is simpler, for SQL-heavy tabular analytics where Dask fits better, and for enterprise SQL ETL pipelines where Spark is the standard choice.