openrlhf-training

Run the train_ppo_ray module with --advantage_estimator group_norm to enable GRPO, which removes the need for a critic model. Add --use_kl_loss and --kl_estimator k3 for the KL loss from the GRPO paper, and optionally --no_advantage_std_norm to disable standard deviation normalization.

PPO uses a critic network for maximum stability but consumes more memory, while GRPO and RLOO are critic-free and about 30% lighter on memory. RLOO adds per-token KL rewards and requires multiple samples per prompt, whereas GRPO uses group reward normalization.

OpenRLHF targets distributed multi-node training of 7B-70B+ models with Ray and vLLM acceleration, running about 2x faster than DeepSpeedChat. TRL suits simpler single-node training, while veRL is ByteDance's framework aimed at very large models like 671B.

A 70B model requires roughly 48 A100 80GB GPUs with the Hybrid Engine enabled, using a vLLM:Actor:Critic ratio of 1:1:1. Without GPU colocation, the same setup needs about 64 dedicated GPUs.

This error occurs because Ray sets CUDA_VISIBLE_DEVICES in a way DeepSpeed does not expect. Fix it by setting the environment variable RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1 before launching training.

Yes, pass a Python file to --remote_rm_url defining a reward_func that receives queries, prompts, and labels and returns rewards, scores, and extra logs. This supports reinforced fine-tuning for tasks like code execution checking or math answer verification.