skills.rest
rocklambros
rocklambrosCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

auditing-rlhf-reward-hacking

Audit RLHF, DPO, and RLAIF checkpoints for reward hacking by comparing reward win-rates with preference win-rates.

2|Updated May 23, 2026
Search Tags:#dpo#rlhf#reward-hacking#model-audit#sycophancy#alignment-tax
One-click install
npx skills add https://github.com/rocklambros/rcs --skill auditing-rlhf-reward-hacking
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: auditing-rlhf-reward-hacking
Source: https://github.com/rocklambros/rcs/tree/main/skills/ml-datasci/auditing-rlhf-reward-hacking
Command: npx skills add https://github.com/rocklambros/rcs --skill auditing-rlhf-reward-hacking

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill audits RLHF, DPO, and RLAIF checkpoints for reward hacking, helping you tell whether a model is truly improving or just gaming the learned reward signal.

Core Features & Use Cases

  • Reward-vs-preference divergence: Compares reward-model win-rate against held-out preference win-rate to expose the main reward-hacking signal.
  • Multi-probe evaluation: Checks length bias, sycophancy, formatting bias, refusal substitution, persuasion over correctness, and boundary exploitation.
  • Promotion gate support: Produces a per-probe verdict table, alignment-tax assessment, and final ship or re-tune recommendation before deployment.

Quick Start

Use this skill to audit an RLHF or DPO model by comparing reward-model and held-out preference results, running the standard probes, and deciding whether the checkpoint is safe to promote.

Frequently Asked Questions about auditing-rlhf-reward-hacking

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I detect reward hacking in an RLHF or DPO checkpoint?▼

Detect reward hacking by comparing reward-model win-rate against held-out preference win-rate. You must evaluate standard failure probes on deployment-representative prompt distributions to distinguish genuine improvement from reward-model gaming.

Does auditing RLHF alignment require separate held-out preference data?▼

Yes, auditing RLHF alignment requires separate held-out preference data. You also need capability benchmarks and probe-specific measurements to accurately identify reward hacking and distinguish true improvement from reward-model gaming.

What probes are used to check DPO checkpoints for sycophancy and length bias?▼

Standard probes check DPO checkpoints for sycophancy, length bias, formatting bias, refusal substitution, persuasion over correctness, and boundary exploitation. These evaluations expose specific reward-hacking behaviors.

Can I use reward hacking audits as a promotion gate before model deployment?▼

Yes, you can use reward hacking audits as a promotion gate. The audit produces a per-probe verdict table, alignment-tax assessment, and final ship or re-tune recommendation before deployment.

How do I measure the alignment tax in RLAIF models?▼

Measure the alignment tax in RLAIF models by evaluating divergence between reward-model win-rate and held-out preference win-rate. Apply standard failure probes on deployment-representative prompt distributions to assess capability degradation.

Why does the reward model win-rate diverge from the held-out preference win-rate?▼

The reward model win-rate diverges from held-out preference win-rate when the model games the learned reward signal rather than genuinely improving. This divergence is the primary indicator of reward hacking in RLHF checkpoints.

Related Skills

Explore More Data & Analytics Skills→
junhyekhjunhyekh
1

rlxp-dr-design

Design domain randomization ranges and schedules for reinforcement learning experiments.

Community
Intermediate
human-avatarhuman-avatar
212

ethics-vendor-review

Evaluate third-party vendors against ethical standards and produce assessments with mitigations.

Official
Intermediate
itsmostafaitsmostafa
24

rlhf

Align language models with human preferences via PPO and DPO workflows.

Community
Advanced
omeragaakbasomeragaakbas

pol-probe

Create falsifiable hypotheses and disposal plans for PoL probes.

Community
Advanced