skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

verl-rl-training

Train LLMs with reinforcement learning using verl's HybridFlow framework.

1|Updated Mar 5, 2026
Search Tags:#reinforcement learning#distributed training#ppo#rlhf#grpo#verl#llm training
One-click install
npx skills add https://github.com/Clay-HHK/claude-skills --skill verl-rl-training-clay-hhk
Or copy as Structured Prompt for Agentâ–¼
Please help me install this Agent Skill.
Skill: verl-rl-training
Source: https://github.com/Clay-HHK/claude-skills/tree/main/verl
Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill verl-rl-training-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires verl, torch, ray, vllm, transformers, and includes references (resource) components.

What problem does it solve? Setting up reinforcement learning post-training for LLMs (RLHF, GRPO, PPO) requires coordinating distributed training backends, rollout engines, and reward functions, which is complex to configure correctly at scale. ## Core Features & Use Cases - Multiple RL Algorithms: Supports PPO, GRPO, DAPO, RLOO, ReMax, REINFORCE++, SPIN, and SPPO with configurable advantage estimators. - Flexible Backends: Swaps between FSDP, FSDP2, and Megatron-LM for training, and vLLM, SGLang, or HuggingFace for rollout generation. - Scale and Modality Support: Trains models from 0.5B to 671B parameters, including vision-language models, LoRA-based RL, and multi-turn tool-calling agents. - Use Case: Train a Qwen2.5-7B model on GSM8K math problems with GRPO by preparing a parquet dataset, defining a rule-based reward function, and launching distributed training across 8 GPUs. ## Quick Start Use the verl skill to set up GRPO training for Qwen2.5-7B on my GSM8K parquet dataset with vLLM rollout across 8 GPUs.

Frequently Asked Questions about verl-rl-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train an LLM with GRPO using verl?â–¼

Prepare a parquet dataset with prompt and reward_model columns, then run python3 -m verl.trainer.main_ppo with algorithm.adv_estimator=grpo and actor_rollout_ref.rollout.n set greater than 1. Enable use_kl_loss with a small kl_loss_coef like 0.001 for stable GRPO training.

What is the difference between GRPO and PPO in verl?â–¼

GRPO is critic-free and uses group-relative advantages, making it suited for sparse-reward tasks like math reasoning. PPO uses adv_estimator=gae with a separate critic model and Generalized Advantage Estimation, which works better for dense reward signals.

Does verl support Megatron-LM for large model training?â–¼

Yes, verl supports Megatron-LM as a training backend for models above 70B parameters with tensor and pipeline parallelism. Install the mbridge package for checkpoint format conversion, and note that Megatron is not supported on AMD GPUs.

How do I fix CUDA out of memory during verl rollout?â–¼

Reduce actor_rollout_ref.rollout.log_prob_micro_batch_size, enable gradient checkpointing, or switch to the fsdp2 strategy with offload_policy set to true. You can also lower vLLM's gpu_memory_utilization from 0.9 to 0.7.

Which vLLM versions are compatible with verl?â–¼

Use vLLM versions between 0.8.2 and 0.12.0, avoiding the 0.7.x series which has known bugs. For vLLM 0.8.x issues, set enforce_eager to true to disable CUDA graphs, and note vLLM 0.11+ requires CUDA 12.1 or newer.

When should I use verl instead of TRL or Axolotl?â–¼

Choose verl for production-scale RL training with multiple algorithms, backend flexibility, and multi-turn tool calling. Use TRL or Axolotl when you only need simpler supervised fine-tuning or DPO without distributed RL infrastructure.

Related Skills

Explore More Education & Research Skills→
thistleknotthistleknot

synthetic-data

Generate synthetic training datasets for LLM fine-tuning with distilabel pipelines.

Community
Advanced
jaredcroxtonjaredcroxton

serving-llms-vllm

Deploy OpenAI-compatible LLM inference servers with vLLM and quantization.

Community
Intermediate
Supporter09Supporter09

weights-and-biases

Tracks ML experiments, sweeps, and model artifacts in Weights & Biases.

Community
Intermediate
qcmuuqcmuu

openrlhf-training

Run distributed OpenRLHF RLHF training on Ray with vLLM acceleration.

Community
Advanced