skills.rest
hanasho744
hanasho744COMMUNITY
Explore all skills by this author →
GitHub

openrlhf-training

Train large language models with distributed PPO, GRPO, RLOO, and DPO using Ray and vLLM.

Updated Mar 23, 2026
Search Tags:#ray#distributed training#ppo#vllm#rlhf#grpo#llm fine-tuning
One-click install
npx skills add https://github.com/hanasho744/codex --skill openrlhf-training-hanasho744
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: openrlhf-training
Source: https://github.com/hanasho744/codex/tree/main/.agents/skills/openrlhf
Command: npx skills add https://github.com/hanasho744/codex --skill openrlhf-training-hanasho744

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires openrlhf, ray, vllm, torch, transformers, deepspeed, and includes references (resource) components.

What problem does it solve? RLHF training of large models (7B-70B+) requires coordinating multiple models (actor, critic, reward, reference) across GPU clusters while keeping generation fast and memory usage manageable. This Skill provides ready-to-run OpenRLHF commands and workflows that handle distributed scheduling with Ray, inference acceleration with vLLM, and GPU sharing via the Hybrid Engine. ## Core Features & Use Cases - Multi-algorithm RLHF training: Run PPO, GRPO, RLOO, REINFORCE++, and DPO from a single framework with algorithm-specific hyperparameter guidance. - Distributed multi-node training: Configure Ray clusters, per-model node/GPU allocation, checkpointing, and fault-tolerant resume for 70B-scale models. - Hybrid Engine GPU sharing: Colocate actor, critic, reward, and reference models with vLLM/DeepSpeed sleep modes to cut GPU requirements by 25-33%. - Custom rewards and agent RLHF: Implement custom reward functions for code generation or math reasoning, and build multi-step agent training loops. - Use Case: Fine-tune a Llama-3-8B model with GRPO on 8 A100 GPUs by submitting a single Ray job with group_norm advantage estimation, vLLM acceleration, and KL loss enabled. ## Quick Start Ask the AI to set up a GRPO training job for a Llama-3-8B model on 8 GPUs using OpenRLHF with vLLM acceleration and colocated models.

Frequently Asked Questions about openrlhf-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a model with GRPO using OpenRLHF?▼

Run the train_ppo_ray module with --advantage_estimator group_norm to enable GRPO, which removes the need for a critic model. Add --use_kl_loss with --kl_estimator k3 for the KL loss term from the GRPO paper, and optionally --no_advantage_std_norm to disable standard deviation normalization.

What is the difference between PPO, GRPO, and RLOO in OpenRLHF?▼

PPO uses a critic network for value estimation and offers the highest stability but the most memory. GRPO normalizes rewards within groups without a critic, while RLOO uses leave-one-out baselines with per-token KL rewards. Critic-free algorithms save roughly 30% memory and run about 20-25% faster than PPO.

OpenRLHF vs TRL vs veRL for RLHF training?▼

OpenRLHF targets distributed multi-node training of 7B-70B+ models with Ray and vLLM acceleration, running about 2x faster than DeepSpeedChat. TRL suits simpler single-node training, while veRL is ByteDance's framework aimed at very large models like 671B parameters.

How many GPUs do I need to train a 70B model with OpenRLHF?▼

A 70B model requires about 48 A100 80GB GPUs with the Hybrid Engine enabled, using a vLLM:Actor:Critic ratio of 1:1:1. Without colocation, the same setup needs 64 dedicated GPUs, so the Hybrid Engine saves roughly 25% of hardware.

Why does OpenRLHF fail with DeepSpeed GPU index out of range?▼

This error occurs because Ray sets CUDA_VISIBLE_DEVICES in a way that conflicts with DeepSpeed device indexing. Fix it by setting the environment variable RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1 before launching the training job.

How do I fix GPU OOM during OpenRLHF PPO training?▼

Enable sleep modes with --vllm_enable_sleep and --deepspeed_enable_sleep so models offload to CPU when idle, and lower --vllm_gpu_memory_utilization to 0.4 or 0.5. If OOM persists, remove --colocate_all_models and allocate separate GPUs per model, or reduce micro batch sizes.

Related Skills

Explore More Software Engineering Skills→
atrawogatrawog

reward

Train reward models with TRL RewardTrainer and LoRA for RLHF pipelines.

Community
Advanced
yo-stevenyo-steven

deployment

Deploy HF LLM checkpoints as OpenAI-compatible endpoints via vLLM, SGLang, or TRT-LLM.

Community
Advanced
Clay-HHKClay-HHK
1

verl-rl-training

Train LLMs with reinforcement learning using verl's HybridFlow framework.

Community
Advanced
azmiariffarisazmiariffaris

fine-tuning-with-trl

Fine-tune LLMs with TRL reinforcement learning to align with human preferences.

Community
Advanced