Modern AI Engineering

Lesson 8.10 · 25 min · video lesson

DPO: Alignment Without the Separate Reward Model

What if the whole reward-model-plus-PPO machinery of RLHF could be replaced by one loss function you train like ordinary fine-tuning?

In short: DPO (Direct Preference Optimization) trains a language model straight from preference pairs (a prompt, a chosen answer and a rejected answer) with a simple classification-style loss. It uses a frozen reference model and a math result showing that the RLHF objective’s optimal policy defines an implicit reward, so no separate reward model and no reinforcement learning loop are needed. It is cheaper and more stable than PPO-based RLHF, but it learns only from fixed data and does not explore.

What you will learn

  • What is RLHF and why do we need it?
  • The problem with RLHF.
  • What is Direct Preference Optimization (DPO)?
  • What is preference data?
  • The key idea behind DPO.
  • The DPO loss function in simple words.
  • How DPO works step-by-step.
  • DPO vs RLHF (PPO).
  • Advantages of DPO.
  • Disadvantages of DPO.

In this lesson

  1. What is RLHF and why do we need it?
  2. The problem with RLHF
  3. What is Direct Preference Optimization?
  4. What is preference data?
  5. The key idea behind DPO
  6. The DPO loss function in simple words
  7. How DPO works step by step
  8. DPO vs RLHF (PPO)
  9. Advantages and disadvantages of DPO
  10. Common mistakes and how to spot them
  11. Practice: try it yourself

Key terms in this lesson

DPO
Direct Preference Optimization: training a policy directly on preference pairs without RL.
Preference pair
A prompt with a chosen (better) and a rejected (worse) response.
Reference model
A frozen copy of the starting (SFT) model used to measure how the policy has changed.
Implicit reward
β · log(π(y|x) / π_ref(y|x)), the reward a policy implicitly assigns to an answer.
Margin
Implicit reward of the chosen answer minus that of the rejected answer.
β (beta)
Controls how far the policy may move from the reference; larger means more conservative.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 8.9 PPO: The Reinforcement Algorithm Behind Instruction Tuning · 8.11 GRPO: Group-Based Preference Optimization Explained →