simpo-training

What problem does it solve?

This Skill describes SimPO, a reference-free preference optimization method to align LLMs, offering stronger performance than DPO while eliminating the need for a reference model.

Core Features & Use Cases

• Reference-free alignment: Train preference models without a reference policy.
• Efficient fine-tuning: Achieve competitive alignment with fewer resources than PPO/DPO.
• Versatile workflows: Used for both baseline LLMs and instruct/chat models with configurable loss and margins.

Quick Start

Launch a SimPO training job for a base model (e.g., Mistral-7B) with a small preference dataset and monitor reward margins.