skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

ray-data

Process large ML datasets with distributed streaming execution across CPU and GPU clusters.

1|Updated Mar 5, 2026
Search Tags:#etl#distributed computing#pytorch#parquet#data preprocessing#ray data#batch inference
One-click install
npx skills add https://github.com/Clay-HHK/claude-skills --skill ray-data-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ray-data
Source: https://github.com/Clay-HHK/claude-skills/tree/main/ray-data
Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill ray-data-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires ray[data], pyarrow, pandas, and includes references (resource) components.

What problem does it solve? Processing datasets larger than memory or scaling data preprocessing and batch inference across a cluster requires distributed infrastructure that is difficult to build manually. This Skill provides guidance for using Ray Data to stream, transform, and load ML data from a single machine to hundreds of nodes. ## Core Features & Use Cases - Distributed Data Loading: Read Parquet, CSV, JSON, and image data from cloud storage (S3, GCS) with streaming execution that handles datasets larger than memory. - Scalable Transformations: Apply vectorized map_batches, filter, and groupby operations across CPU workers, with optional GPU acceleration for image and tensor preprocessing. - ML Framework Integration: Feed data directly into Ray Train via dataset shards, or convert to PyTorch and TensorFlow iterators for training loops. - Use Case: Run distributed batch inference by loading a model once per worker, applying it with map_batches over a Parquet dataset, and writing predictions back to S3. ## Quick Start Use the ray-data skill to build a pipeline that reads Parquet files from S3, preprocesses them in batches, and feeds them into distributed PyTorch training.

Frequently Asked Questions about ray-data

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I process large datasets with Ray Data in Python?▼

Use ray.data.read_parquet or read_csv to load data, then apply map_batches for vectorized transformations. Ray Data uses streaming execution, so datasets larger than memory are processed in batches without loading everything at once.

How to use Ray Data with PyTorch training?▼

Convert a Ray Dataset with ds.to_torch(label_column="label", batch_size=32) to get an iterable of tensor batches, or use iter_torch_batches. With Ray Train, pass datasets to TorchTrainer and access shards via ray.train.get_dataset_shard.

Ray Data vs Pandas vs Spark for data processing?▼

Use Pandas for small data under 1GB on a single machine, Dask or Spark for SQL-style tabular operations and enterprise ETL. Ray Data fits ML workloads needing streaming execution, GPU-accelerated transforms, and tight integration with distributed training.

Does Ray Data support GPU-accelerated preprocessing?▼

Yes, pass num_gpus=1 to map_batches and use a callable that moves data to CUDA inside the function. This is commonly used for image preprocessing, where benchmarks show roughly 5,000 images per second on one GPU versus 1,000 on CPU.

When should I not use Ray Data?▼

Avoid Ray Data for small datasets under 1GB where Pandas is simpler, and for SQL-heavy enterprise ETL where Spark is better suited. It is designed for ML pipelines, batch inference, and multi-modal loading rather than general-purpose analytics.

Why is map_batches faster than map in Ray Data?▼

map_batches applies vectorized operations over batches of rows using NumPy or pandas, making it 10-100x faster than row-by-row map. Tune batch_size upward for better throughput while staying within memory limits.

Related Skills

Explore More Data & Analytics Skills→
Clay-HHKClay-HHK
1

ray-data

Process large ML datasets with distributed streaming execution across CPU and GPU clusters.

Community
Intermediate
OpenLAIROpenLAIR
1.0k

ray-train

Orchestrate distributed machine learning training across multiple nodes and clusters.

Official
Advanced
organvm-i-theoriaorganvm-i-theoria

ml-experiment-tracker

Track ML experiments, version datasets, and manage model registries with MLflow, W&B, and DVC.

Official
Intermediate
qinyan-aiqinyan-ai
48

dask

Scale pandas and NumPy workflows to datasets that exceed memory.

Community
Advanced