Modern AI Engineering

Lesson 8.9 · 26 min · video lesson

PPO: The Reinforcement Algorithm Behind Instruction Tuning

One overly eager update can wreck a policy that took days to train; how does PPO let a model learn from its own experience without taking that kind of leap?

In short: PPO (Proximal Policy Optimization) is a reinforcement learning algorithm that improves a policy in small, safe steps. It compares how likely an action is under the new policy versus the old one (the probability ratio), multiplies by the advantage (how much better than expected the action was), and clips the ratio to a narrow band such as [0.8, 1.2] so no single update can move the policy too far. It is simple, stable and was the standard RL algorithm in RLHF for language models.

What you will learn

  • What is Reinforcement Learning?
  • What is a Policy?
  • The problem with simple policy updates.
  • What is Proximal Policy Optimization (PPO)?
  • The key idea behind PPO: Clipping.
  • The PPO objective function in simple words.
  • How PPO works step-by-step.
  • PPO in Large Language Models (RLHF).
  • Advantages of PPO.
  • Disadvantages of PPO.

In this lesson

  1. What is reinforcement learning?
  2. What is a policy?
  3. The problem with simple policy updates
  4. What is PPO, and the key idea: clipping
  5. The PPO objective in simple words
  6. How PPO works step by step
  7. PPO in large language models (RLHF)
  8. Advantages and disadvantages of PPO
  9. Worked example, step by step
  10. Practice: try it yourself

Key terms in this lesson

Policy
A function giving the probability of each action in a state; in LLMs, the model itself.
Advantage
How much better an action did than expected: return minus the value estimate.
Value function
A model V(s) predicting the expected future reward from a state.
Probability ratio
π_new(a|s) / π_old(a|s), how much an action’s probability changed.
Clipping
Limiting the ratio to [1−ε, 1+ε] so updates stay small.
TRPO
Trust Region Policy Optimization: PPO’s predecessor with a hard KL constraint.
GAE
Generalised advantage estimation: a lower-noise way to compute advantages.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 8.8 RLHF: Aligning LLMs with Human Preferences · 8.10 DPO: Alignment Without the Separate Reward Model →