skills.rest
hanasho744
hanasho744COMMUNITY
Explore all skills by this author →
GitHub

simpo-training

Train LLMs with reference-free SimPO preference optimization using alignment-handbook configs.

Updated Mar 23, 2026
Search Tags:#fine-tuning#preference optimization#simpo#post-training#llm alignment#dpo alternative#reference-free
One-click install
npx skills add https://github.com/hanasho744/codex --skill simpo-training-hanasho744
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/hanasho744/codex/tree/main/.agents/skills/orchestra-simpo
Command: npx skills add https://github.com/hanasho744/codex --skill simpo-training-hanasho744

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, datasets, trl, accelerate, and includes references (resource) components.

What problem does it solve? Aligning LLMs with human preferences typically requires DPO or PPO setups that demand a separate reference model, extra memory, and complex configuration. This Skill guides SimPO training, a reference-free preference optimization method that outperforms DPO on AlpacaEval 2.0 while using fewer resources. ## Core Features & Use Cases - Reference-Free Preference Training: Run SimPO on chosen/rejected preference pairs without loading a reference model, using HuggingFace alignment-handbook and DeepSpeed ZeRO-3. - Hyperparameter Guidance: Get recommended beta, gamma-beta ratio, learning rate, and SFT weight values by model size (7B to 70B) and task type (chat, math, code). - Dataset Preparation: Use ready-made configs for UltraFeedback, HH-RLHF, HelpSteer, and math preference datasets, plus filtering and mixing strategies. - Use Case: Fine-tune Mistral 7B on UltraFeedback preference pairs with a single accelerate launch command, then troubleshoot loss divergence or capability forgetting using the documented fixes. ## Quick Start Ask the assistant to set up SimPO training for Mistral 7B on the UltraFeedback dataset and generate the training config and launch command.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a model with SimPO instead of DPO?▼

Install the HuggingFace alignment-handbook, create a YAML config specifying beta, gamma_beta_ratio, and your preference dataset, then launch with accelerate using the run_simpo.py script and a DeepSpeed ZeRO-3 config. No reference model is needed, unlike DPO.

What is the difference between SimPO and DPO for preference alignment?▼

SimPO optimizes preferences directly from policy log probabilities without a reference model, while DPO normalizes against a reference model's outputs. SimPO reports +6.4 points over DPO on AlpacaEval 2.0 and is more memory-efficient since it runs fewer model forward passes.

What learning rate should I use for SimPO training on a 7B model?▼

Use 3e-7 to 5e-7 for 7B models, with 5e-7 as the standard default for general chat tasks. Lower it to 3e-7 for math or code reasoning tasks, and reduce further if you observe loss divergence during training.

Why is my SimPO training loss diverging?▼

Loss divergence usually means the learning rate or beta is too high. Reduce learning_rate from 5e-7 to 3e-7 and lower beta from 2.0 to 1.0, then monitor the reward margin between chosen and rejected responses.

How do I prevent my instruct model from forgetting capabilities during SimPO training?▼

Add SFT regularization by setting sft_weight between 0.05 and 0.1 in your config. This mixes a cross-entropy loss on chosen responses into the SimPO loss, preserving instruction-following abilities at a small cost to alignment strength.

What GPU hardware is required for SimPO training?▼

A 7B model fits on one A100 40GB with DeepSpeed ZeRO-3, an 8B model needs two A100 40GB, and a 70B model requires eight A100 80GB. Enable BF16, gradient checkpointing, and Flash Attention 2 to reduce memory usage.

Related Skills

Explore More Education & Research Skills→
big4council-progbig4council-prog

simpo-training

Train language models from chosen/rejected preference pairs without a reference model.

Community
Advanced
thinking-machines-labthinking-machines-lab
4.0k

weights

Downloads, merges LoRA adapters, and publishes models to HuggingFace Hub.

Official
Intermediate
kwasi-cpukwasi-cpu

fine-tuning-with-trl

Fine-tune LLMs with TRL using SFT, DPO, PPO, and GRPO.

Community
Advanced
eddieluengeddielueng
1

fine-tuning-with-trl

Fine-tune language models with TRL for SFT, DPO, and PPO/GRPO workflows.

Community
Advanced