Modern AI Engineering

Lesson 8.11 · 25 min · video lesson

GRPO: Group-Based Preference Optimization Explained

Instead of training a whole second network to guess how good an answer “should” be, what if we just asked the model the same question eight times and compared the answers with each other?

In short: GRPO (Group Relative Policy Optimization) is a reinforcement learning method for language models that removes PPO’s value model. For each prompt it samples a group of answers, scores them, and uses each answer’s reward relative to the group’s mean (divided by the group’s standard deviation) as its advantage. It keeps PPO’s clipped update and a KL penalty to a reference model. Introduced with DeepSeekMath and used for DeepSeek-R1, it is a popular choice for training reasoning models with verifiable rewards.

What you will learn

  • What is GRPO?
  • Why do we need GRPO?
  • The problem with PPO.
  • How does GRPO work?
  • Step-by-step example.
  • The GRPO objective in simple words.
  • Advantages of GRPO.
  • Practical things to keep in mind.
  • When to use GRPO.
  • Conclusion.

In this lesson

  1. What is GRPO?
  2. Why do we need GRPO? The problem with PPO
  3. How does GRPO work?
  4. Step-by-step example
  5. The GRPO objective in simple words
  6. PPO vs GRPO
  7. Advantages of GRPO
  8. Practical things to keep in mind
  9. Going one level deeper
  10. Practice: try it yourself
  11. When to use GRPO, and conclusion

Key terms in this lesson

GRPO
Group Relative Policy Optimization: PPO-style RL that uses group-normalised rewards instead of a value model.
Group
Several answers sampled from the policy for the same prompt.
Group-relative advantage
(reward − group mean) / group standard deviation.
Value model (critic)
A network PPO uses to predict expected reward; GRPO removes it.
Verifiable reward
A reward computed by a program, such as checking a final answer or running tests.
Reference model
A frozen copy of the starting policy used for the KL penalty.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 8.10 DPO: Alignment Without the Separate Reward Model · 9.1 Chain-of-Thought Prompting: Making Models Reason Step by Step →