skills.rest
Orchestra-Research
Orchestra-ResearchOFFICIAL
Explore all skills by this author →
GitHub

slime-rl-training

Configures RL post-training for LLMs using Megatron-LM training with SGLang rollout generation.

13.0k|930|Updated Nov 3, 2025
Search Tags:#reinforcement learning#sglang#rlhf#grpo#post-training#megatron-lm#llm training
One-click install
npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill slime-rl-training-orchestra-research
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: slime-rl-training
Source: https://github.com/Orchestra-Research/AI-research-SKILLs/tree/main/06-post-training/slime
Command: npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill slime-rl-training-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires sglang-router, ray, torch, transformers, and includes references (resource) components.

What problem does it solve? Setting up reinforcement learning post-training for large language models requires coordinating distributed training, high-throughput rollout generation, and data management, which is complex to configure correctly across frameworks. ## Core Features & Use Cases - GRPO and PPO Training: Run group-relative advantage estimation and other RL algorithms with Megatron-LM parallelism (TP, PP, DP, SP) and SGLang-based rollout. - Asynchronous and Multi-Turn Training: Overlap rollout and training for higher throughput, or train agents with tool use via custom generate functions. - Flexible Data Buffer: Manage prompts and rollout samples with custom filtering, off-policy buffering, and custom reward functions. - Use Case: Train a Qwen3-4B reasoning model on math problems by preparing JSONL prompt-label data, sourcing a pre-configured model script, and launching GRPO training with KL loss across 8 GPUs. ## Quick Start Ask the AI to set up a GRPO training run with slime for a Qwen3 model using your JSONL prompt dataset, including data preparation, model configuration, and launch commands.

Frequently Asked Questions about slime-rl-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train an LLM with GRPO using slime?▼

Prepare a JSONL dataset with prompt and label fields, source a pre-configured model script from scripts/models/, then launch train.py with --advantage-estimator grpo, --use-kl-loss, and your GPU allocation flags. Ensure rollout_batch_size times n_samples_per_prompt equals global_batch_size times num_steps_per_rollout.

slime vs verl vs torchforge for RL post-training?▼

Choose slime for Megatron-LM native training with SGLang inference and custom data generation workflows, especially for GLM models. Use verl for flexible backend swapping, torchforge for PyTorch-native abstractions, or miles for enterprise-grade stability features.

Which models does slime support for RL training?▼

slime supports GLM-4.5/4.6/4.7, Qwen3 (4B, 8B, 30B-A3B, MoE), DeepSeek V3/V3.1/R1, Llama 3 (8B, 70B), Kimi K2, and Moonlight-16B. Each model has pre-configured scripts in the scripts/models/ directory.

Can slime do multi-turn agent training with tool calls?▼

Yes, slime supports multi-turn agentic training through a custom generate function passed via --custom-generate-function-path. The function handles tool execution loops and must set loss masks so tool responses are excluded from loss computation.

Why does slime training OOM during the backward pass?▼

CUDA OOM during training typically results from large batch sizes or long sequences. Enable gradient checkpointing with --recompute-activations, reduce --micro-batch-size to 1, enable --sequence-parallel, or lower --global-batch-size.

Does slime async training work with colocated GPUs?▼

No, colocated mode is not supported with async training in slime. When using train_async.py, allocate separate GPUs for training and rollout instead of passing the --colocate flag.

Related Skills

Explore More Education & Research Skills→
ovachieverovachiever
52

grpo-rl-training

Apply GRPO/RL training with TRL to optimize language models.

Community
Advanced
MLT-OSSMLT-OSS
14

evaluating-llms-harness

Evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.

Official
Advanced
castform-aicastform-ai
49

verify-environment

Validate Castform training environments by interpreting scorecards and reward variance.

Official
Advanced
tadod12tadod12

verl-rl-training

Configure scalable Verl RL training workflows for LLMs across vLLM and Megatron backends.

Community
Advanced