skills.rest
azaanaliraza
azaanalirazaCOMMUNITY
Explore all skills by this author →
GitHub

simpo-training

Train LLMs with reference-free SimPO preference optimization using HuggingFace alignment-handbook configs.

Updated Apr 18, 2026
Search Tags:#fine-tuning#huggingface#preference optimization#simpo#deepspeed#llm alignment#dpo alternative
One-click install
npx skills add https://github.com/azaanaliraza/operarius --skill simpo-training-azaanaliraza
Or copy as Structured Prompt for Agentâ–¼
Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/azaanaliraza/operarius/tree/main/src-tauri/bin/hermes/optional-skills/mlops/simpo
Command: npx skills add https://github.com/azaanaliraza/operarius --skill simpo-training-azaanaliraza

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, datasets, trl, accelerate, and includes references (resource) components.

What problem does it solve? Aligning LLMs with human preferences typically requires DPO or PPO setups that demand a separate reference model, doubling memory usage and complicating training pipelines. SimPO removes the reference model entirely while outperforming DPO, making preference alignment simpler and more compute-efficient. ## Core Features & Use Cases - Reference-Free Preference Training: Run SimPO training on base or instruct models (Mistral 7B, Llama 3 8B) using YAML configs with the HuggingFace alignment-handbook and DeepSpeed ZeRO-3. - Hyperparameter Guidance: Get recommended values for beta, gamma-beta ratio, learning rate, and SFT weight by model size and task type (general chat, math, code). - Dataset & Troubleshooting References: Access detailed guides on preference dataset formats, quality filtering, loss functions (sigmoid vs hinge), and fixes for loss divergence, OOM, and capability forgetting. - Use Case: Fine-tune Llama 3 8B Instruct on argilla/ultrafeedback-binarized-preferences-cleaned with beta 2.5 and sft_weight 0.1 to improve AlpacaEval performance without a reference model. ## Quick Start Ask the assistant to generate a SimPO training config and launch command for your chosen base model and preference dataset.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a model with SimPO preference optimization?â–¼

Install the HuggingFace alignment-handbook, create a YAML config specifying your model, dataset_mixer, beta, and gamma_beta_ratio, then launch with accelerate using the DeepSpeed ZeRO-3 config and scripts/run_simpo.py. A 7B model typically trains on a single A100 40GB.

What is the difference between SimPO and DPO?â–¼

SimPO is reference-free, optimizing directly on the policy's average log probabilities with a target margin, while DPO requires a frozen reference model to normalize log probability ratios. SimPO reports +6.4 points over DPO on AlpacaEval 2.0 with lower memory and compute.

What beta and learning rate should I use for SimPO?â–¼

For 7B models, use learning rate 3e-7 to 5e-7, beta 2.0 to 2.5, and gamma_beta_ratio 0.5. Use lower learning rates around 3e-7 for math or code tasks, and higher beta up to 5.0 for stronger preference signals.

What dataset format does SimPO training require?â–¼

Preference datasets need prompt, chosen, and rejected fields, such as HuggingFaceH4/ultrafeedback_binarized or argilla/ultrafeedback-binarized-preferences-cleaned. Multiple datasets can be mixed with weights in the dataset_mixer config section.

Why does SimPO training loss diverge or forget capabilities?â–¼

Loss divergence usually means the learning rate is too high, so reduce it from 5e-7 to 3e-7 or lower beta. If the model forgets capabilities, add SFT regularization by setting sft_weight to 0.1, which is recommended for instruct models.

What GPU hardware is needed for SimPO training?â–¼

A 7B model trains on one A100 40GB with DeepSpeed ZeRO-3, an 8B model needs two A100 40GB, and a 70B model requires eight A100 80GB. BF16 mixed precision, gradient checkpointing, and Flash Attention 2 reduce memory usage.

Related Skills

Explore More Software Engineering Skills→
box755box755

simpo-training

Train an LLM with SimPO on chosen/rejected pairs without a reference model.

Community
Intermediate
yanochka11yanochka11

evaluating-llms-harness

Benchmark large language models with lm-evaluation-harness across standardized metrics.

Community
Advanced
SethyPagnaSethyPagna

fine-tuning-with-trl

Fine-tune language models with TRL for post-training alignment workflows.

Community
Advanced
handsomelong922handsomelong922

openrlhf-training

Orchestrate distributed RLHF training for PPO, GRPO, RLOO, and DPO workflows.

Community
Advanced