skills.rest
kaminocorp
kaminocorpOFFICIAL
Explore all skills by this author →
Website𝕏@kaminocorpoGitHub

simpo-training

Train LLMs with reference-free SimPO preference optimization using alignment-handbook configs.

1|Updated Mar 12, 2026
Search Tags:#fine-tuning#training#preference optimization#simpo#deepspeed#llm alignment#dpo alternative
One-click install
npx skills add https://github.com/kaminocorp/hermes-alpha-hunter --skill simpo-training-kaminocorp
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: simpo-training
Source: https://github.com/kaminocorp/hermes-alpha-hunter/tree/main/skills/mlops/training/simpo
Command: npx skills add https://github.com/kaminocorp/hermes-alpha-hunter --skill simpo-training-kaminocorp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, datasets, trl, accelerate, and includes references (resource) components.

What problem does it solve? Aligning LLMs with human preferences typically requires DPO or PPO setups that demand a separate reference model, extra GPU memory, and complex configuration. This Skill provides ready-to-use SimPO training workflows that achieve better alignment results without a reference model, reducing compute cost and setup complexity. ## Core Features & Use Cases - Reference-Free Preference Training: Train Mistral, Llama 3, and other models on preference pairs using SimPO's sigmoid or hinge loss with no reference model forward passes. - Tuned Hyperparameter Guidance: Model-size-specific recommendations for learning rate, beta, gamma-beta ratio, and SFT weight, plus troubleshooting for loss divergence, capability forgetting, and OOM errors. - Dataset Preparation References: Detailed guidance on preference data formats, quality filtering, dataset mixing, and synthetic pair generation. - Use Case: Fine-tune Llama 3 8B Instruct on cleaned UltraFeedback preferences with a single accelerate launch command and a YAML config, preserving instruction-following ability via SFT regularization. ## Quick Start Ask the agent to train Mistral 7B with SimPO on the UltraFeedback dataset using the provided training config and DeepSpeed ZeRO-3.

Frequently Asked Questions about simpo-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a model with SimPO instead of DPO?▼

Install the HuggingFace alignment-handbook, create a YAML config specifying your model, preference dataset, beta, and gamma_beta_ratio, then launch with accelerate using the run_simpo.py script and a DeepSpeed ZeRO-3 config. No reference model is needed, unlike DPO.

What is the difference between SimPO and DPO for LLM alignment?▼

SimPO is reference-free, optimizing directly on the policy's average log probabilities with a target margin, while DPO normalizes against a frozen reference model. SimPO reports +6.4 points on AlpacaEval 2.0 over DPO and uses less memory since it skips reference model forward passes.

What learning rate should I use for SimPO training on a 7B model?▼

Use 3e-7 to 5e-7 for 7B models, with 5e-7 as the standard default for general chat tasks. Lower it to 3e-7 for math or code reasoning tasks, and reduce further if you observe loss divergence during training.

Why is my SimPO training loss diverging?▼

Loss divergence usually comes from a learning rate that is too high or an overly large beta. Reduce learning rate from 5e-7 to 3e-7 and lower beta from 2.0 to 1.0, then monitor the reward margin between chosen and rejected responses.

How do I prevent my instruct model from forgetting capabilities during preference training?▼

Add SFT regularization by setting sft_weight between 0.05 and 0.2 in your config. This mixes cross-entropy loss on chosen responses into the SimPO loss, preserving instruction-following ability at a small cost to alignment strength.

What GPU hardware is required for SimPO training?▼

A 7B model fits on one A100 40GB with DeepSpeed ZeRO-3, an 8B model needs two A100 40GB, and a 70B model requires eight A100 80GB. Enable BF16 mixed precision, gradient checkpointing, and Flash Attention 2 to reduce memory usage.

Related Skills

Explore More Data & Analytics Skills→
box755box755

simpo-training

Train an LLM with SimPO on chosen/rejected pairs without a reference model.

Community
Intermediate
yanochka11yanochka11

evaluating-llms-harness

Benchmark large language models with lm-evaluation-harness across standardized metrics.

Community
Advanced
Orchestra-ResearchOrchestra-Research
13.0k

fine-tuning-with-trl

Fine-tune LLMs with TRL using SFT, DPO, PPO, GRPO, and reward model training.

Official
Advanced
hung-phanhung-phan
1

distributed-grpo

Scale GRPO/PPO reinforcement learning training across multiple GPUs with DeepSpeed.

Community
Advanced