fine-tuning-with-trl

What problem does it solve?

This Skill enables the fine-tuning of Large Language Models (LLMs) to align their outputs with human preferences and instructions, improving their helpfulness and safety.

Core Features & Use Cases

• Reinforcement Learning from Human Feedback (RLHF): Implement full RLHF pipelines using SFT, Reward Models, and PPO/GRPO.
• Direct Preference Optimization (DPO): Align models with preferences directly from chosen/rejected pairs without a separate reward model.
• Reward Model Training: Train models to score the quality of LLM generations.
• Use Case: You have a base LLM that generates factually correct but sometimes unhelpful or biased responses. Use this Skill to fine-tune it using preference data so it becomes more aligned with desired conversational behavior.

Quick Start

Use the fine-tuning-with-trl skill to perform Supervised Fine-Tuning on a Qwen2.5-0.5B model using the Capybara dataset.