skills.rest
80portisfound
80portisfoundCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

simpo-training

Train LLMs with reference-free SimPO preference optimization using HuggingFace alignment-handbook configs.

Updated May 3, 2026
Search Tags:#fine-tuning#huggingface#preference optimization#simpo#deepspeed#llm alignment#dpo alternative
One-click install
npx skills add https://github.com/80portisfound/vibe-learning --skill simpo-training-80portisfound
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/80portisfound/vibe-learning/tree/main/packages/hermes/optional-skills/mlops/simpo
Command: npx skills add https://github.com/80portisfound/vibe-learning --skill simpo-training-80portisfound

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, datasets, trl, accelerate, and includes references (resource) components.

What problem does it solve? Aligning LLMs with human preferences typically requires DPO or PPO setups that demand a separate reference model, extra GPU memory, and complex tuning. This Skill guides you through SimPO, a reference-free preference optimization method that outperforms DPO (+6.4 points on AlpacaEval 2.0) while being simpler and more efficient to train. ## Core Features & Use Cases - Reference-Free Preference Training: Train on chosen/rejected preference pairs without loading a reference model, cutting memory and compute requirements. - Ready-Made Training Workflows: Pre-built configs for Mistral 7B base, Llama 3 8B instruct, and reasoning-intensive math/code tasks with tuned beta, gamma, and learning rate values. - Troubleshooting & Tuning Guides: Reference docs covering loss functions (sigmoid vs hinge), hyperparameter selection by model size, and preference dataset preparation. - Use Case: You have an instruct model and a cleaned UltraFeedback preference dataset. Use this Skill to configure and launch a SimPO training run with DeepSpeed ZeRO-3 on a single node, adding light SFT regularization to preserve instruction-following capabilities. ## Quick Start Ask the AI to set up a SimPO training run for a Mistral 7B base model using the UltraFeedback dataset with the recommended default hyperparameters.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a model with SimPO preference optimization?▼

Install the HuggingFace alignment-handbook, prepare a YAML config with your model, preference dataset, beta, and gamma_beta_ratio, then launch with accelerate using a DeepSpeed ZeRO-3 config and the run_simpo.py script.

SimPO vs DPO: which preference optimization method should I use?▼

SimPO is simpler and more efficient because it needs no reference model, and it outperforms DPO by 6.4 points on AlpacaEval 2.0. Choose DPO when you need a conservative reference-model baseline for comparison.

What learning rate should I use for SimPO training on a 7B model?▼

Use 3e-7 to 5e-7 for 7B models, with 5e-7 as the standard default. Lower it to 3e-7 for reasoning-intensive tasks like math or code, and reduce further if the loss diverges.

What dataset format does SimPO training require?▼

SimPO needs preference pairs with prompt, chosen, and rejected fields, such as HuggingFaceH4/ultrafeedback_binarized. Alternative field names like question, winner, and loser are auto-detected, and multiple datasets can be mixed with weights.

Why is my SimPO training loss diverging?▼

Loss divergence usually means the learning rate or beta is too high. Reduce learning rate from 5e-7 to 3e-7 and lower beta from 2.0 to 1.0, then monitor the reward margin between chosen and rejected responses.

How much GPU memory does SimPO training need?▼

A 7B model fits on one A100 40GB with DeepSpeed ZeRO-3, an 8B model needs two A100 40GB, and a 70B model needs eight A100 80GB. Enable gradient checkpointing and Flash Attention 2 to reduce memory further.

Related Skills

Explore More Software Engineering Skills→
box755box755

simpo-training

Train an LLM with SimPO on chosen/rejected pairs without a reference model.

Community
Intermediate
thinking-machines-labthinking-machines-lab
4.0k

weights

Downloads, merges LoRA adapters, and publishes models to HuggingFace Hub.

Official
Intermediate
junhyekhjunhyekh
1

rlxp-audit-results

Audit reinforcement-learning improvement claims for validity across RLXP experiments.

Community
Advanced
adm-humanerdadm-humanerd

fine-tuning-with-trl

Automate end-to-end RLHF fine-tuning of LLMs with TRL.

Community
Advanced