ray-train

Wrap your training loop in a train_func, call train.torch.prepare_model on your model, and pass it to TorchTrainer with a ScalingConfig setting num_workers and use_gpu=True. Ray handles device placement, coordination, and metric aggregation automatically.

Ray Train is best for multi-node orchestration, integrated hyperparameter tuning via Ray Tune, and fault tolerance. Use Accelerate for simpler single-node multi-GPU, Lightning for high-level abstractions and callbacks, and DeepSpeed for maximum performance with more complex setup.

Yes, Ray Train provides TransformersTrainer for HuggingFace integration. You keep your standard TrainingArguments and Trainer code, and Ray distributes it across workers specified in ScalingConfig, such as 16 workers across multiple nodes.

On AWS, define a cluster.yaml with node types and run ray up to launch it. On Kubernetes, install the KubeRay operator via Helm and apply a RayCluster manifest specifying head and worker GPU resources.

Save checkpoints periodically with train.report and Checkpoint.from_directory, then restore state at startup via train.get_checkpoint. Set FailureConfig(max_failures=3) in RunConfig so Ray automatically retries failed workers and resumes from the last checkpoint.

Run ray status to verify nodes and GPUs are registered. If not, restart the head node with ray start --head --port=6379, then reconnect workers using ray start --address=<head-ip>:6379, ensuring ports 6379, 8265, and 10001 are open.