skills.rest
hanasho744
hanasho744COMMUNITY
Explore all skills by this author →
GitHub

grpo-rl-training

Implement GRPO reinforcement learning fine-tuning with TRL and custom reward functions.

Updated Mar 23, 2026
Search Tags:#fine-tuning#reinforcement learning#lora#reward functions#trl#grpo#llm training
One-click install
npx skills add https://github.com/hanasho744/codex --skill grpo-rl-training-hanasho744
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: grpo-rl-training
Source: https://github.com/hanasho744/codex/tree/main/.agents/skills/grpo-rl-training
Command: npx skills add https://github.com/hanasho744/codex --skill grpo-rl-training-hanasho744

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires transformers, trl, datasets, peft, torch.

What problem does it solve? Fine-tuning language models to follow strict output formats or solve verifiable tasks is difficult with supervised methods alone, and GRPO training involves subtle pitfalls like misread loss curves, mode collapse, and poorly designed reward functions. ## Core Features & Use Cases - Complete GRPO Workflow: Step-by-step guidance covering dataset preparation, reward function design, training configuration, and deployment with TRL. - Reward Function Library: Over 20 ready-to-adapt reward functions for correctness, format, length, and style objectives, plus presets for math, code, summarization, and Q&A tasks. - Production Template: A runnable training script using Qwen 2.5 1.5B with LoRA, three reward functions, and memory-optimized hyperparameters. - Use Case: Train a model to answer math problems in a strict <reasoning>/<answer> XML format by combining an incremental format reward with an exact-match correctness reward. ## Quick Start Use the grpo-rl-training skill to set up a GRPO training run with TRL that teaches my model to output reasoning and answers in XML tags.

Frequently Asked Questions about grpo-rl-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a model with GRPO using TRL?▼

Prepare a dataset with chat-formatted prompts, define 3-5 reward functions returning float scores, then configure GRPOConfig with num_generations of 8-16 and pass everything to GRPOTrainer. The included template provides a complete working script using Qwen 2.5 1.5B with LoRA.

What reward functions should I use for GRPO training?▼

Combine 3-5 reward functions covering different signals: a correctness reward weighted highest around 2.0, a format reward around 0.5, and optional length or style rewards. Test each function independently before training to verify scoring logic.

Why does GRPO loss increase during training?▼

Rising loss is expected in GRPO because the loss measures KL divergence from the initial policy, not prediction error. Monitor reward and reward_std metrics instead to track actual learning progress.

When should I use GRPO instead of DPO or SFT?▼

Use GRPO for verifiable tasks with objective reward signals like math or code, or to enforce output formats without preference pairs. Choose SFT for simple supervised tasks and DPO when you already have high-quality preference data.

How do I fix GRPO running out of GPU memory?▼

Reduce num_generations, lower per_device_train_batch_size, enable gradient checkpointing, or use the adamw_8bit optimizer. Unsloth with 4-bit loading and vLLM generation can also cut memory use and speed training 2-3x.

Related Skills

Explore More Software Engineering Skills→
JoaquinCampoJoaquinCampo
3

gsm8k-eval

Extract final numeric answers from GSM8K model outputs using official ground-truth formats.

Community
Advanced
GitnappGitnapp

peft-fine-tuning

Configure LoRA and QLoRA adapters for memory-efficient LLM fine-tuning.

Community
Advanced
shuff57shuff57

weights-and-biases

Automate ML experiment logging and model management with Weights & Biases.

Community
Advanced
KappTech88KappTech88

fine-tuning-with-trl

Align language models with human preferences using TRL fine-tuning methods.

Community
Advanced