skills.rest
hanasho744
hanasho744COMMUNITY
Explore all skills by this author →
GitHub

miles-rl-training

Configures RL training for large MoE models with FP8, INT4, and speculative decoding.

Updated Mar 23, 2026
Search Tags:#reinforcement learning#speculative decoding#fp8#sglang#megatron#moe#int4
One-click install
npx skills add https://github.com/hanasho744/codex --skill miles-rl-training-hanasho744
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: miles-rl-training
Source: https://github.com/hanasho744/codex/tree/main/.agents/skills/orchestra-miles
Command: npx skills add https://github.com/hanasho744/codex --skill miles-rl-training-hanasho744

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires sglang-router, ray, torch, transformers, and includes references (resource) components.

What problem does it solve? Training trillion-parameter MoE models with reinforcement learning often fails due to FP8 instability, train-inference mismatch, and slow rollout throughput. This Skill provides guidance for using miles, a production fork of slime, to run stable large-scale RL post-training. ## Core Features & Use Cases - Low-Precision Training: Unified FP8 pipelines and INT4 quantization-aware training that fit 1TB models on a single H200 machine. - Train-Inference Alignment: Rollout Routing Replay (R3) records expert routing during SGLang inference and replays it in Megatron training for bit-wise MoE alignment. - Speculative RL: EAGLE speculative decoding with online MTP draft training delivers 25-40% faster rollouts. - Use Case: You are post-training DeepSeek V3 with GRPO and hitting NaN losses under FP8. This Skill walks you through enabling block scaling, R3, and TIS correction to stabilize training. ## Quick Start Ask the assistant to set up a miles GRPO training run for a Qwen3-MoE model with FP8 and speculative decoding enabled.

Frequently Asked Questions about miles-rl-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train large MoE models with RL using miles?▼

Run miles train.py with your HuggingFace checkpoint, set --advantage-estimator grpo, and configure expert-model-parallel-size for MoE parallelism. Enable FP8 block scaling via the NVTE_FP8_BLOCK_SCALING_FP32_SCALES environment variable for stability.

miles vs slime vs verl for RL training?▼

miles is a production fork of slime adding FP8/INT4 training, R3 routing replay, and speculative RL. Choose slime for research-grade flexibility, verl for backend swapping, or torchforge for PyTorch-native abstractions.

Does miles support FP8 and INT4 training?▼

Yes, miles provides unified end-to-end FP8 for training and inference, plus INT4 quantization-aware training. INT4 reduces VRAM roughly 3.1x, enabling 671B models on a single H200 machine.

Why does FP8 RL training collapse with NaN losses?▼

FP8 collapse usually stems from quantization discrepancy between training and inference. Enable block scaling, reduce learning rate to 5e-7, and turn on Rollout Routing Replay so MoE expert routing matches between SGLang and Megatron.

How do I speed up RL rollouts with speculative decoding?▼

Enable EAGLE speculative decoding with --sglang-speculative-algorithm EAGLE and provide a draft model path. Add online MTP training flags to keep the draft model aligned, yielding 25-40% faster rollouts.

Which models does miles support for RL training?▼

miles supports DeepSeek R1/V3, Qwen 2/2.5/3 including MoE variants, Llama 3/4, Gemma 2/3, GLM 4.x, and MiniMax M2. Full MoE support covers DeepSeek, Qwen MoE, and MiniMax families.

Related Skills

Explore More Software Engineering Skills→
thinking-machines-labthinking-machines-lab
4.0k

datasets

Construct and normalize ML datasets across JSONL, HuggingFace, and conversation formats.

Official
Intermediate
yo-stevenyo-steven

deployment

Deploy HF LLM checkpoints as OpenAI-compatible endpoints via vLLM, SGLang, or TRT-LLM.

Community
Advanced
davila7davila7
30.5k

openrlhf-training

Trains large language models with PPO, GRPO, RLOO, and DPO using Ray and vLLM.

Community
Advanced
tadod12tadod12

verl-rl-training

Configure scalable Verl RL training workflows for LLMs across vLLM and Megatron backends.

Community
Advanced