skills.rest
kaminocorp
kaminocorpOFFICIAL
Explore all skills by this author →
Website𝕏@kaminocorpoGitHub

grpo-rl-training

Implements GRPO reinforcement learning fine-tuning of language models with TRL and custom reward functions.

1|Updated Mar 12, 2026
Search Tags:#fine-tuning#reinforcement learning#lora#reward functions#trl#grpo#llm training
One-click install
npx skills add https://github.com/kaminocorp/hermes-alpha-hunter --skill grpo-rl-training-kaminocorp
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: grpo-rl-training
Source: https://github.com/kaminocorp/hermes-alpha-hunter/tree/main/skills/mlops/training/grpo-rl-training
Command: npx skills add https://github.com/kaminocorp/hermes-alpha-hunter --skill grpo-rl-training-kaminocorp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires transformers, trl, datasets, peft, torch.

What problem does it solve? Fine-tuning language models to follow strict output formats or excel at verifiable tasks like math and code is difficult with supervised methods alone. This Skill provides a complete workflow for Group Relative Policy Optimization (GRPO) training with TRL, covering dataset preparation, reward function design, training configuration, and debugging. ## Core Features & Use Cases - Reward Function Library: 20+ ready-to-use reward functions for correctness, format compliance, length control, and style, with guidance on combining 3-5 rewards per task. - Production Training Template: A complete training script using Qwen 2.5 1.5B with LoRA, flash attention, and memory-optimized GRPOConfig settings. - Training Diagnostics: Explains expected loss behavior (loss increases are normal), reward_std monitoring, and fixes for mode collapse, OOM errors, and format non-compliance. - Use Case: Train a model to answer math problems in a strict <reasoning>/<answer> XML format by combining correctness and incremental format rewards on the GSM8K dataset. ## Quick Start Ask the agent to set up GRPO training on GSM8K using the basic template with format and correctness reward functions.

Frequently Asked Questions about grpo-rl-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a model with GRPO using TRL?▼

Prepare a dataset with chat-formatted prompts, define 3-5 reward functions returning float scores, configure GRPOConfig with num_generations of 8-16, and pass everything to GRPOTrainer. Start with a small model like Qwen2.5-1.5B-Instruct and LoRA for memory efficiency.

What reward functions should I use for GRPO training?▼

Combine 3-5 reward functions covering correctness (weight 2.0), format compliance (0.5-1.0), and length or style (0.1-0.5). Test each function independently before training, and use incremental rewards that give partial credit for partial format compliance.

When should I use GRPO instead of DPO or SFT?▼

Use GRPO for verifiable tasks with objective correctness signals, format enforcement, or multi-objective optimization without preference data. Use SFT for simple supervised tasks and DPO when you already have high-quality preference pairs.

Why does loss increase during GRPO training?▼

Rising loss is expected because GRPO loss measures KL divergence from the initial policy as the model diverges to optimize rewards. Monitor reward, reward_std, and KL metrics instead of loss to judge training progress.

How do I fix mode collapse in GRPO training?▼

Mode collapse appears when reward_std drops toward zero and completions become identical. Increase num_generations for more within-group diversity, add diversity-penalizing reward functions, and verify reward functions are not overly harsh.

Can GRPO training run on a small GPU?▼

Yes, use per_device_train_batch_size of 1 with gradient accumulation, LoRA with rank 16, the adamw_8bit optimizer, and num_generations of 4-8. Unsloth provides 2-3x speedup with 4-bit quantization for further memory savings.

Related Skills

Explore More Data & Analytics Skills→
JoaquinCampoJoaquinCampo
3

gsm8k-eval

Extract final numeric answers from GSM8K model outputs using official ground-truth formats.

Community
Advanced
GitnappGitnapp

peft-fine-tuning

Configure LoRA and QLoRA adapters for memory-efficient LLM fine-tuning.

Community
Advanced
NVIDIANVIDIA
49

sop-ddm-finetuning

Fine-tune DDM-Net temporal boundary detectors on annotated SOP video datasets.

Official
Advanced
KappTech88KappTech88

fine-tuning-with-trl

Align language models with human preferences using TRL fine-tuning methods.

Community
Advanced