Modern AI Engineering

Lesson 8.8 · 27 min · video lesson

RLHF: Aligning LLMs with Human Preferences

If people can easily tell which of two answers is better but cannot write down a formula for “good answer”, how do we train a model to give better answers?

In short: RLHF (reinforcement learning from human feedback) aligns a language model with human preferences in three stages: supervised fine-tuning on good examples, training a reward model on human comparisons, and reinforcement learning (usually PPO) to maximise that reward. A KL penalty keeps the model close to its starting point so it does not exploit the reward model’s flaws, a failure called reward hacking.

What you will learn

  • What is RLHF
  • Why we need RLHF
  • The Big Picture
  • Stage 1: Supervised Fine-Tuning (SFT)
  • Stage 2: Training the Reward Model
  • Stage 3: RL Fine-Tuning with PPO
  • The KL Penalty
  • Putting It All Together
  • Reward Hacking
  • Common Mistakes
  • Best Practices
  • Quick Summary

In this lesson

  1. What is RLHF?
  2. Why we need RLHF
  3. The big picture
  4. Stage 1: supervised fine-tuning (SFT)
  5. Stage 2: training the reward model
  6. Stage 3: RL fine-tuning with PPO
  7. The KL penalty
  8. Putting it all together
  9. Reward hacking
  10. Common mistakes and best practices
  11. Worked example, step by step
  12. Practice: try it yourself
  13. Quick summary

Key terms in this lesson

RLHF
Reinforcement learning from human feedback: optimising a model against a reward learned from human preferences.
Reward model
A model that gives a scalar score predicting human preference for a response.
Policy
In RLHF, the language model being trained, viewed as choosing tokens (actions).
Reference model
A frozen copy of the SFT model used to compute the KL penalty.
KL penalty
A cost proportional to how far the policy’s distribution moves from the reference model.
Reward hacking
Raising the proxy reward without improving real quality.
Value model (critic)
A model PPO uses to estimate expected reward, so advantages can be computed.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 8.7 InstructGPT: Teaching GPT-3 to Follow Instructions · 8.9 PPO: The Reinforcement Algorithm Behind Instruction Tuning →