skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

simpo-training

Train LLMs with reference-free SimPO preference optimization using alignment-handbook configs.

1|Updated Mar 5, 2026
Search Tags:#fine-tuning#preference optimization#simpo#llm alignment#dpo alternative#reference-free#alignment-handbook
One-click install
npx skills add https://github.com/Clay-HHK/claude-skills --skill simpo-training-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/Clay-HHK/claude-skills/tree/main/simpo
Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill simpo-training-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, datasets, trl, accelerate, and includes references (resource) components.

What problem does it solve? Aligning LLMs with preference data typically requires DPO's reference model, doubling memory and compute. This Skill guides SimPO training, a reference-free alternative that outperforms DPO (+6.4 points on AlpacaEval 2.0) with simpler, faster single-node training. ## Core Features & Use Cases - Reference-Free Preference Training: Configure and launch SimPO runs via HuggingFace alignment-handbook with DeepSpeed ZeRO-3, Flash Attention 2, and BF16. - Hyperparameter Guidance: Model-size-specific recommendations for beta, gamma_beta_ratio, learning rate, and SFT weight, plus troubleshooting for loss divergence, capability forgetting, and OOM. - Dataset Preparation: Preference data formats, quality filtering, dataset mixing (UltraFeedback, HH-RLHF, math preferences), and synthetic pair generation. - Use Case: Fine-tune Llama 3 8B Instruct on cleaned UltraFeedback preferences with beta 2.5 and light SFT regularization to improve alignment while preserving instruction-following capabilities. ## Quick Start Train Mistral 7B with SimPO on the UltraFeedback dataset using the alignment-handbook and DeepSpeed ZeRO-3 configuration.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a model with SimPO instead of DPO?▼

Install the HuggingFace alignment-handbook, then launch scripts/run_simpo.py with a YAML config specifying beta, gamma_beta_ratio, and your preference dataset. SimPO needs no reference model, so training uses less memory than DPO.

SimPO vs DPO: which preference optimization method should I use?▼

SimPO is simpler and more efficient since it requires no reference model, and it outperforms DPO by 6.4 points on AlpacaEval 2.0. Choose DPO when you need a conservative reference-model baseline for comparison.

What learning rate and beta should I use for SimPO on a 7B model?▼

For 7B models, use learning rate 3e-7 to 5e-7, beta 2.0 to 2.5, and gamma_beta_ratio 0.5. Use lower learning rates around 3e-7 for math and code reasoning tasks.

Why is my SimPO training loss diverging?▼

Loss divergence usually means the learning rate or beta is too high. Reduce learning rate from 5e-7 to 3e-7 and lower beta from 2.0 to 1.0 to stabilize training.

How do I prevent capability forgetting during SimPO fine-tuning?▼

Add SFT regularization by setting sft_weight to 0.1 in the config, which mixes cross-entropy loss on chosen responses with the SimPO loss. This is recommended when fine-tuning instruct or chat models.

What GPUs are required for SimPO training on 7B to 70B models?▼

A 7B model fits on one A100 40GB with DeepSpeed ZeRO-3, an 8B model needs two A100 40GB, and a 70B model requires eight A100 80GB. BF16, gradient checkpointing, and Flash Attention 2 reduce memory usage.

Related Skills

Explore More Education & Research Skills→
kaminocorpkaminocorp
1

simpo-training

Train LLMs with reference-free SimPO preference optimization using alignment-handbook configs.

Official
Advanced
RachasumanthRachasumanth

inference-deploy

Deploy trained machine learning models for inference using vLLM, TGI, Ollama, and llama.cpp.

Community
Advanced
junhyekhjunhyekh
1

rlxp-audit-results

Audit reinforcement-learning improvement claims for validity across RLXP experiments.

Community
Advanced
adm-humanerdadm-humanerd

fine-tuning-with-trl

Automate end-to-end RLHF fine-tuning of LLMs with TRL.

Community
Advanced