ray-data

What problem does it solve?

This Skill addresses the challenge of processing large datasets efficiently for machine learning, enabling distributed data preprocessing and scalable pipelines.

Core Features & Use Cases

• Distributed Data Processing: Handles datasets larger than memory across multiple nodes.
• Multi-modal Support: Works with various data formats including Parquet, CSV, JSON, and images.
• Framework Integration: Seamlessly integrates with Ray Train, PyTorch, and TensorFlow.
• Use Case: Use this Skill to preprocess terabytes of image and text data for a deep learning model training job, distributing the workload across a cluster of machines.

Quick Start

Use the ray-data skill to read all parquet files from 's3://my-bucket/data/' and then apply a lowercasing transformation to the 'text' column.