skills.rest
hanasho744
hanasho744COMMUNITY
Explore all skills by this author →
GitHub

fine-tuning-with-trl

Fine-tune language models with TRL using SFT, DPO, PPO, GRPO, and reward model training.

Updated Mar 23, 2026
Search Tags:#fine-tuning#dpo#trl#rlhf#sft#grpo#reward model
One-click install
npx skills add https://github.com/hanasho744/codex --skill fine-tuning-with-trl-hanasho744
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: fine-tuning-with-trl
Source: https://github.com/hanasho744/codex/tree/main/.agents/skills/orchestra-trl-fine-tuning
Command: npx skills add https://github.com/hanasho744/codex --skill fine-tuning-with-trl-hanasho744

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires trl, transformers, datasets, peft, accelerate, torch, and includes references (resource) components.

What problem does it solve? Aligning language models with human preferences requires complex RLHF pipelines involving supervised fine-tuning, reward modeling, and reinforcement learning, which are difficult to implement correctly from scratch. ## Core Features & Use Cases - Supervised Fine-Tuning (SFT): Train base models on instruction-following datasets with chat templates, packing, and LoRA support. - Preference Alignment with DPO: Align models using chosen/rejected preference pairs without training a separate reward model, with 10+ loss variants including IPO, hinge, and robust DPO. - Online RL with PPO and GRPO: Optimize policies against reward models or custom reward functions, with GRPO offering memory-efficient training. - Reward Model Training: Train Bradley-Terry reward models on preference data for use in full RLHF pipelines. - Use Case: Take a base Qwen model, run SFT on instruction data, train a reward model on UltraFeedback preferences, then apply PPO to produce a human-aligned chat model. ## Quick Start Ask the AI to fine-tune a Qwen model with DPO on a preference dataset using TRL and save the aligned model to an output directory.

Frequently Asked Questions about fine-tuning-with-trl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I fine-tune an LLM with DPO using TRL?▼

Use DPOTrainer with a preference dataset containing prompt, chosen, and rejected fields. Configure DPOConfig with beta (default 0.1), learning rate around 5e-7, and pass the model, tokenizer, and dataset, then call trainer.train().

What is the difference between SFT, DPO, PPO, and GRPO in TRL?▼

SFT trains on prompt-completion pairs for instruction following. DPO aligns with preference pairs without a reward model. PPO uses a trained reward model for full RL control. GRPO runs online RL with lower memory by sampling multiple completions per prompt.

How much GPU memory does TRL training require?▼

For a 7B model, SFT needs about 16GB with LoRA, DPO about 24GB since it stores a reference model, PPO about 40GB for policy plus reward model, and GRPO about 24GB. LoRA, gradient checkpointing, and gradient accumulation reduce memory further.

Why does DPO training run out of memory?▼

DPO holds both policy and reference models in memory, so long sequences and large batches cause OOM. Reduce per_device_train_batch_size to 1, lower max_length, increase gradient_accumulation_steps, or enable gradient checkpointing.

Can I use LoRA with TRL trainers?▼

Yes, pass a peft LoraConfig to SFTTrainer via the peft_config argument, and the same approach works with other TRL trainers. This significantly reduces VRAM usage for methods like SFT, DPO, and reward modeling.

When should I use Axolotl or Unsloth instead of TRL?▼

Use Axolotl for YAML-driven training configuration, Unsloth for fast LoRA training, or plain HuggingFace Trainer for basic fine-tuning without RL. TRL is the right choice when you need RLHF, preference alignment, or reward model training.

Related Skills

Explore More Software Engineering Skills→
big4council-progbig4council-prog

simpo-training

Train language models from chosen/rejected preference pairs without a reference model.

Community
Advanced
itallstartedwithaideaitallstartedwithaidea
31

ml-model-integration

Select and deploy HuggingFace Hub models with task-first filtering and evaluation.

Community
Advanced
ginkidaginkida
20

llm-finetuning

Fine-tunes LLMs with LoRA/QLoRA for domain-specific NLP tasks.

Community
Advanced
HugoAlmeidaMoreiraHugoAlmeidaMoreira

axolotl

Fine-tune large language models with YAML configs and LoRA adapters.

Community
Advanced