skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

torchforge-rl-training

Implements PyTorch-native agentic RL training with GRPO, Monarch, TorchTitan, and vLLM.

1|Updated Mar 5, 2026
Search Tags:#reinforcement learning#pytorch#distributed training#vllm#grpo#llm training#torchtitan
One-click install
npx skills add https://github.com/Clay-HHK/claude-skills --skill torchforge-rl-training-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: torchforge-rl-training
Source: https://github.com/Clay-HHK/claude-skills/tree/main/torchforge
Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill torchforge-rl-training-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch>=2.9.0, torchtitan>=0.2.0, vllm, monarch, and includes references (resource) components.

What problem does it solve? Setting up reinforcement learning training for LLMs typically requires wiring together distributed training, inference, and weight synchronization infrastructure by hand. This Skill provides guidance for using torchforge, Meta's PyTorch-native RL library that separates algorithm logic from infrastructure so you can focus on reward functions and loss design. ## Core Features & Use Cases - GRPO and SFT Training Workflows: Step-by-step configuration and launch instructions for math reasoning training with group-relative advantages, including reward function definition and monitoring checklists. - Custom Loss Functions: Implement new RL algorithms (GRPO, DAPO, CISPO, GSPO, SAPO variants) as isolated PyTorch modules without touching distributed infrastructure. - Multi-GPU Scaling: Configure tensor parallelism and distributed services across GPUs or SLURM clusters using Monarch actors, TorchTitan FSDP training, and vLLM inference. - Use Case: Train Qwen2.5-7B on GSM8K with GRPO across 3 GPUs by writing a YAML config and a reward function, then launching with a single command. ## Quick Start Ask the assistant to set up a GRPO training run with torchforge for a math reasoning dataset on 3 GPUs, including the YAML config and reward function.

Frequently Asked Questions about torchforge-rl-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run GRPO training with torchforge?▼

Create a YAML config specifying the model, dataset, GRPO hyperparameters (n_samples, clip bounds, beta), and service GPU allocations, then launch with python -m apps.grpo.main --config your_config.yaml. You need at least 3 GPUs for trainer, generator, and reference model services.

What is torchforge and when should I use it?▼

torchforge is Meta's PyTorch-native RL library that separates algorithm code from distributed infrastructure using Monarch actors, TorchTitan training, and vLLM inference. Use it for clean RL abstractions and fast algorithm experimentation; choose verl or miles for production stability instead.

How many GPUs does torchforge need for training?▼

SFT requires at least 2 GPUs (trainer and generator), while GRPO needs 3 or more for trainer, generator, and reference model. You can reduce requirements by running the reference model on CPU or colocating services.

Does torchforge support AMD GPUs with ROCm?▼

Yes, torchforge provides a dedicated ROCm installation script at ./scripts/install_rocm.sh for AMD GPU environments. The standard install.sh handles PyTorch nightly and dependencies for CUDA setups.

Why does torchforge training hit CUDA OOM during generation?▼

OOM in vLLM generation usually comes from large n_samples or long sequences. Reduce grpo n_samples from 8 to 4, lower seq_len from 4096 to 2048, or decrease vLLM gpu_memory_utilization to 0.7.

How do I fix policy collapse in GRPO training?▼

Policy collapse shows as entropy dropping to zero with stalled rewards. Increase the KL penalty beta from 0.1 to 0.2, add an entropy coefficient around 0.01, or reduce the learning rate to stabilize training.

Related Skills

Explore More Software Engineering Skills→
dongzhuoyaodongzhuoyao

experiment-env-installer

Install reproducible Python environments for ML experiments with uv across compute backends.

Community
Advanced
NVIDIANVIDIA
3.7k

deployment

Serve quantized or unquantized LLM checkpoints as OpenAI-compatible endpoints using vLLM, SGLang, or TRT-LLM.

Official
Advanced
plurigridplurigrid
60

fokker-planck-analyzer

Compare empirical weight distributions to Gibbs distributions for Langevin training convergence.

Official
Advanced
hung-phanhung-phan
1

distributed-grpo

Scale GRPO/PPO reinforcement learning training across multiple GPUs with DeepSpeed.

Community
Advanced