Reinforcement Learning from Human Feedback (RLHF)

RLHF is a training technique that aligns LLM outputs with human preferences. It was popularized by OpenAI's InstructGPT and is used in GPT-4o, Claude 4, and many other frontier models.

The RLHF Pipeline

Stage 1: Supervised Fine-Tuning (SFT)

The pretrained model is fine-tuned on human-written demonstrations of desired behavior.

Stage 2: Reward Model Training

A reward model is trained on human preference comparisons:

Humans are shown two model outputs and pick the better one
The reward model learns to predict which output humans prefer
This creates a scalable proxy for human judgment

Stage 3: Policy Optimization

The original language model (now the "policy") is optimized using the reward model as a signal via PPO (Proximal Policy Optimization):

The policy generates text
The reward model scores it
PPO updates the policy to increase reward
A KL penalty prevents the model from diverging too far

Limitations

Expensive — Requires large-scale human annotation
Reward hacking — The model may optimize for reward without genuine improvement
Mode collapse — Can reduce output diversity
Complex engineering — PPO is notoriously difficult to stabilize

Alternatives

Direct Preference Optimization (DPO) — Simpler alternative without explicit reward model
Rejection Sampling — Generate many outputs, keep the best
Constitutional AI — Self-critique and revision (used by Anthropic)

Comparison

Aspect	RLHF	DPO
Reward model	Required	Not needed
Complexity	High (PPO)	Low (direct loss)
Stability	Requires tuning	More stable
Performance	Strong	Comparable

Reinforcement Learning from Human Feedback