skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

openrlhf-training

Train large language models with PPO, GRPO, RLOO, and DPO using Ray and vLLM.

1|Updated Mar 5, 2026
Search Tags:#fine-tuning#ray#distributed training#ppo#vllm#rlhf#grpo
One-click install
npx skills add https://github.com/Clay-HHK/claude-skills --skill openrlhf-training-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: openrlhf-training
Source: https://github.com/Clay-HHK/claude-skills/tree/main/openrlhf
Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill openrlhf-training-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires openrlhf, ray, vllm, torch, transformers, deepspeed, and includes references (resource) components.

What problem does it solve? RLHF training of large models (7B-70B+) requires coordinating actor, critic, reward, and reference models across many GPUs, which is slow and memory-intensive with naive setups. This Skill provides distributed RLHF training workflows built on Ray, vLLM, and DeepSpeed ZeRO-3 with GPU sharing to reduce idle time and hardware requirements. ## Core Features & Use Cases - Multi-Algorithm RL Training: Run PPO, GRPO, RLOO, REINFORCE++, and DPO from a single framework by switching the advantage estimator flag. - Hybrid Engine GPU Sharing: Colocate all models on shared GPUs with vLLM and DeepSpeed sleep modes, cutting GPU needs by roughly 25% for 70B-scale training. - Custom Rewards and Agent RLHF: Plug in custom reward functions for code or math tasks, or run multi-step agent training with environment feedback loops. - Use Case: Fine-tune a Llama-3-8B model with GRPO on a math dataset using a custom answer-checking reward function across an 8-GPU Ray cluster, with checkpointing and W&B monitoring. ## Quick Start Set up a Ray cluster and launch PPO training on my 8-GPU node using the Llama-3-8b SFT model and a reward model, with vLLM acceleration and model colocation enabled.

Frequently Asked Questions about openrlhf-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a model with GRPO in OpenRLHF?▼

Run the standard PPO training entry point with --advantage_estimator group_norm to enable GRPO. Add --use_kl_loss and --kl_estimator k3 for the KL loss from the GRPO paper, and omit the critic model since GRPO does not need one.

What is the difference between PPO, GRPO, and RLOO in OpenRLHF?▼

PPO uses a critic network for maximum stability but consumes more memory. GRPO normalizes rewards within sample groups without a critic, while RLOO uses leave-one-out baselines with per-token KL. Critic-free methods save roughly 30% memory on 7B models.

OpenRLHF vs TRL vs veRL for RLHF training?▼

OpenRLHF targets distributed multi-node training of 7B-70B+ models with Ray and vLLM acceleration. TRL suits single-node training with a simpler API, while veRL is ByteDance's framework aimed at very large models like 671B.

How many GPUs do I need to train a 70B model with OpenRLHF?▼

A 70B model needs about 48 A100 80GB GPUs with the Hybrid Engine enabled, using a 1:1:1 ratio of vLLM, actor, and critic resources. Without colocation, plan for around 64 dedicated GPUs.

Why does OpenRLHF fail with DeepSpeed GPU index out of range?▼

This error occurs when Ray sets CUDA_VISIBLE_DEVICES in a way DeepSpeed cannot map. Fix it by setting the environment variable RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1 before launching training.

Can I use a custom reward function instead of a reward model?▼

Yes, pass a Python file via --remote_rm_url implementing a reward_func that receives queries, prompts, and labels and returns rewards, scores, and extra logs. This supports reinforced fine-tuning for tasks like code execution testing or math answer checking.

Related Skills

Explore More Education & Research Skills→
atrawogatrawog

reward

Train reward models with TRL RewardTrainer and LoRA for RLHF pipelines.

Community
Advanced
VYRE-StudiosVYRE-Studios

serving-llms-vllm

Serve LLMs with vLLM using PagedAttention and continuous batching.

Official
Advanced
plurigridplurigrid
60

fokker-planck-analyzer

Compare empirical weight distributions to Gibbs distributions for Langevin training convergence.

Official
Advanced
quiznatquiznat

fine-tuning-with-trl

Train language models with TRL RLHF workflows for preference alignment.

Community
Advanced