skills.rest
Orchestra-Research
Orchestra-ResearchOFFICIAL
Explore all skills by this author →
GitHub

constitutional-ai

Trains harmless language models through self-critique, revision, and RL from AI feedback.

13.0k|930|Updated Nov 3, 2025
Search Tags:#ppo#self-critique#trl#constitutional ai#safety alignment#rlaif#reward model
One-click install
npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill constitutional-ai-orchestra-research
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: constitutional-ai
Source: https://github.com/Orchestra-Research/AI-research-SKILLs/tree/main/07-safety-alignment/constitutional-ai
Command: npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill constitutional-ai-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires transformers, torch, trl, and includes references (resource) components.

What problem does it solve? Aligning language models to be harmless typically requires expensive human-labeled preference data for harmful outputs. Constitutional AI removes that bottleneck by letting the model critique and revise its own responses against a written set of principles, then training with AI-generated preferences (RLAIF) instead of human labels. ## Core Features & Use Cases - Supervised Self-Critique Phase: Generate initial responses, critique them against a constitution, revise, and fine-tune on the revised outputs using TRL's SFTTrainer. - RLAIF Phase: Sample response pairs, collect AI preference judgments based on the constitution, train a reward model with RewardTrainer, and run PPO-based RL training. - Chain-of-Thought Critique: Produce step-by-step reasoning about helpfulness, honesty, and harmlessness for transparent alignment decisions. - Use Case: A team fine-tuning a 7B assistant wants to reduce toxic and harmful outputs without hiring human annotators. They define a four-principle constitution, run self-critique and revision on red-team prompts, then train a reward model from AI preferences and complete PPO training on two A100 GPUs. ## Quick Start Use the constitutional-ai skill to set up a two-phase training pipeline that critiques and revises harmful prompt responses against my constitution, then trains a reward model from AI preferences for RLAIF.

Frequently Asked Questions about constitutional-ai

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a harmless AI model without human labels?▼

Constitutional AI replaces human labels with AI feedback in two phases. First, the model critiques and revises its own responses against a written constitution for supervised fine-tuning. Second, AI-generated preference comparisons train a reward model for PPO-based RLAIF.

What is the difference between RLAIF and RLHF?▼

RLAIF uses AI-generated preference judgments, making it scalable and free of human labeling costs, while RLHF relies on human preferences that are more accurate but expensive. Constitutional AI uses RLAIF with a constitution guiding the AI evaluator.

How do I implement Constitutional AI with TRL?▼

Use SFTTrainer to fine-tune on critique-revised responses, then RewardTrainer on AI preference pairs, and finally PPOTrainer with the trained reward model. The workflow requires transformers, torch, and trl as dependencies.

Why does my aligned model refuse too many requests?▼

Excessive refusal happens when the constitution overweights harmlessness. Add a principle preferring responses that engage thoughtfully and explain concerns rather than refusing, then regenerate critiques and revisions.

What hardware is needed for Constitutional AI training?▼

The supervised phase for a 7B model needs one A100 40GB GPU, while the RL phase needs two A100 40GB GPUs to hold both the policy and reward model. BF16 mixed precision is recommended, and a single node suffices for most cases.

When should I use DPO instead of Constitutional AI?▼

Use DPO or SimPO when you already have human preference data, since they train directly on preferences without a reward model. Constitutional AI fits when you lack human labels and want scalable, principle-driven safety training.

Related Skills

Explore More Education & Research Skills→
erfanzarerfanzar
27

axolotl

Automate end-to-end language model fine-tuning workflows with Axolotl.

Community
Advanced
tianhao909tianhao909
1

llamaguard

Filter LLM input and output across six safety categories using Meta's LlamaGuard model.

Community
Advanced
AevonixAevonix

fine-tuning-with-trl

Fine-tune language models with TRL for SFT, DPO, and PPO/GRPO alignment.

Official
Intermediate
intelligent-alphaintelligent-alpha

case-study

Synthesize investment case studies from decision logs and diligence trails.

Official
Advanced