skills.rest
hanasho744
hanasho744COMMUNITY
Explore all skills by this author →
GitHub

verl-rl-training

Train LLMs with reinforcement learning using verl's HybridFlow framework.

Updated Mar 23, 2026
Search Tags:#reinforcement learning#distributed training#ppo#rlhf#grpo#verl#llm training
One-click install
npx skills add https://github.com/hanasho744/codex --skill verl-rl-training-hanasho744
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: verl-rl-training
Source: https://github.com/hanasho744/codex/tree/main/.agents/skills/orchestra-verl
Command: npx skills add https://github.com/hanasho744/codex --skill verl-rl-training-hanasho744

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires verl, torch, ray, vllm, transformers, and includes references (resource) components.

What problem does it solve? Setting up reinforcement learning post-training for large language models requires coordinating distributed training backends, rollout engines, reward functions, and algorithm configurations, which is error-prone without structured guidance. ## Core Features & Use Cases - RL Algorithm Workflows: Step-by-step configurations for GRPO, PPO/GAE, RLOO, REINFORCE++, and DAPO with critic-free or critic-based setups. - Backend Flexibility: Guidance for FSDP, FSDP2, and Megatron-LM training backends plus vLLM and SGLang rollout engines, scaling from 0.5B to 671B parameter models. - Troubleshooting References: Detailed solutions for OOM errors, training instability, weight sync delays, and vLLM version conflicts. - Use Case: Train a math reasoning model on GSM8K using GRPO with Qwen2.5-7B, a custom boxed-answer reward function, and 8-sample rollouts per prompt. ## Quick Start Ask the assistant to set up a GRPO training run with verl on a GSM8K parquet dataset using Qwen2.5-7B and vLLM rollout on 8 GPUs.

Frequently Asked Questions about verl-rl-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train an LLM with GRPO using verl?▼

Prepare a parquet dataset with prompt and reward_model columns, define a reward function, then launch verl.trainer.main_ppo with algorithm.adv_estimator=grpo and rollout n greater than 1. Enable use_kl_loss with a small kl_loss_coef such as 0.001 for stable training.

What is the difference between GRPO and PPO in verl?▼

GRPO is critic-free and estimates advantages from grouped samples, suiting sparse-reward tasks like math reasoning. PPO uses GAE with a separate critic model, which works better for dense rewards but requires more GPU memory.

Does verl support Megatron-LM for large model training?▼

Yes, verl supports Megatron-LM as a training backend for models above 70B parameters with tensor and pipeline parallelism. Install the mbridge package for checkpoint conversion, and note Megatron is not supported on AMD GPUs.

Why does verl run out of memory during rollout?▼

Rollout OOM typically comes from large log_prob_micro_batch_size or high vLLM gpu_memory_utilization. Reduce the micro batch size, enable gradient checkpointing, lower gpu_memory_utilization to 0.7, or switch to FSDP2 with CPU offloading.

Which vLLM versions are compatible with verl?▼

verl works with vLLM versions 0.8.2 through 0.12.0. Avoid vLLM 0.7.x due to known bugs, and note that vLLM 0.11 and later require CUDA 12.1 or newer.

When should I use TRL instead of verl for LLM training?▼

Use TRL or Axolotl when you only need simple supervised fine-tuning or DPO without distributed RL infrastructure. verl is designed for production-scale RL training with flexible backend swapping and multi-turn agentic rollouts.

Related Skills

Explore More Software Engineering Skills→
issdandavisissdandavis
6

scbe-training-pair-authoring

Convert SCBE source material into structured JSONL training pairs with provenance.

Community
Advanced
OpenLAIROpenLAIR
1.0k

miles-rl-training

Configure miles for large MoE RL training with FP8/INT4 and train-inference alignment.

Official
Advanced
tianhao909tianhao909
1

tensorboard

Visualize machine learning training metrics and model graphs with TensorBoard.

Community
Intermediate
OpenLAIROpenLAIR
1.0k

fine-tuning-with-trl

Automates end-to-end LLM fine-tuning workflows with TRL and RLHF techniques.

Official
Advanced