What problem does it solve? Aligning language models to be harmless typically requires expensive human-labeled preference data for harmful outputs. This Skill implements Anthropic's Constitutional AI method, which achieves safety alignment through self-critique, revision, and AI-generated preferences without human labels. ## Core Features & Use Cases - Supervised Self-Critique Phase: Generate responses, critique them against a written constitution, revise them, and fine-tune with TRL's SFTTrainer. - RLAIF Phase: Build preference pairs from AI evaluations, train a reward model with RewardTrainer, and run PPO-based RL training. - Chain-of-Thought Critique: Add step-by-step reasoning to critiques for transparent, explainable safety decisions. - Use Case: You want to reduce harmful outputs from a fine-tuned 7B model but lack budget for human annotation. Define a constitution of principles, let the model critique and revise its own responses, then train a reward model from AI preferences to complete alignment. ## Quick Start Ask the AI to set up a Constitutional AI training pipeline that generates self-critiques and revisions for a set of prompts, then trains a reward model and runs PPO with TRL.
