skills.rest
Orchestra-Research
Orchestra-ResearchOFFICIAL
Explore all skills by this author →
GitHub

simpo-training

Train LLMs with reference-free SimPO preference optimization using HuggingFace alignment-handbook configs.

13.0k|930|Updated Nov 3, 2025
Search Tags:#fine-tuning#preference optimization#simpo#post-training#llm alignment#dpo alternative#reference-free
One-click install
npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill simpo-training-orchestra-research
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/Orchestra-Research/AI-research-SKILLs/tree/main/06-post-training/simpo
Command: npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill simpo-training-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, datasets, trl, accelerate, and includes references (resource) components.

What problem does it solve? Aligning LLMs with human preferences typically requires DPO or PPO setups that demand a separate reference model, doubling memory usage and complicating training pipelines. SimPO removes the reference model entirely while outperforming DPO on benchmarks like AlpacaEval 2.0. ## Core Features & Use Cases - Reference-Free Preference Training: Run SimPO training via accelerate and DeepSpeed ZeRO-3 with YAML configs for models like Mistral 7B and Llama 3 8B. - Hyperparameter Guidance: Detailed references for beta, gamma-beta ratio, learning rate, and SFT weight selection by model size and task type. - Dataset Preparation: Guidance on preference data formats, quality filtering, dataset mixing, and synthetic pair generation. - Use Case: Fine-tune Llama 3 8B Instruct on cleaned UltraFeedback preference pairs with a single command, adding light SFT regularization to preserve instruction-following capabilities. ## Quick Start Train Mistral 7B on the UltraFeedback preference dataset using SimPO with the provided YAML config and accelerate launch command.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a model with SimPO instead of DPO?▼

Install the HuggingFace alignment-handbook, prepare a YAML config with beta, gamma_beta_ratio, and learning_rate, then launch with accelerate using a DeepSpeed ZeRO-3 config and the run_simpo.py script. No reference model is needed, unlike DPO.

SimPO vs DPO: which preference optimization method should I use?▼

SimPO is simpler and more efficient since it requires no reference model and outperforms DPO by +6.4 points on AlpacaEval 2.0. Choose DPO when you need an established reference-model baseline or more conservative training behavior.

What learning rate should I use for SimPO training on 7B models?▼

Use 3e-7 to 5e-7 for 7B-8B models, with 5e-7 as the standard default. Lower it to 3e-7 for reasoning-intensive tasks like math or code, and reduce further if loss diverges during training.

Why does my SimPO model forget its capabilities after training?▼

Capability forgetting happens when pure preference optimization overwrites prior instruction-following behavior. Add SFT regularization by setting sft_weight to 0.1-0.2, which mixes cross-entropy loss on chosen responses into the total loss.

What GPUs are required for SimPO training on 7B models?▼

A single A100 40GB with DeepSpeed ZeRO-3 handles 7B models, while 8B models need 2× A100 40GB and 70B models need 8× A100 80GB. Enable BF16, gradient checkpointing, and Flash Attention 2 to reduce memory usage.

What dataset format does SimPO preference training require?▼

Datasets need prompt, chosen, and rejected fields containing preference pairs. Popular ready-to-use options include HuggingFaceH4/ultrafeedback_binarized, argilla/ultrafeedback-binarized-preferences-cleaned, and Anthropic/hh-rlhf.

Related Skills

Explore More Education & Research Skills→
agtktIDagtktID
1

simpo-training

Trains language models with reference-free SimPO preference optimization using preference pair datasets.

Community
Intermediate
yanochka11yanochka11

evaluating-llms-harness

Benchmark large language models with lm-evaluation-harness across standardized metrics.

Community
Advanced
SethyPagnaSethyPagna

fine-tuning-with-trl

Fine-tune language models with TRL for post-training alignment workflows.

Community
Advanced
quiznatquiznat

fine-tuning-with-trl

Train language models with TRL RLHF workflows for preference alignment.

Community
Advanced