Modern AI Engineering

Lesson 8.7 · 27 min · video lesson

InstructGPT: Teaching GPT-3 to Follow Instructions

How did a 1.3-billion-parameter model end up preferred by people over the 175-billion-parameter GPT-3 it was built from?

In short: InstructGPT (OpenAI, 2022) turned GPT-3 from a text-continuation engine into a model that follows instructions. It used three steps: supervised fine-tuning on human-written answers, a reward model trained on human rankings of answers, and reinforcement learning with PPO against that reward, kept close to the original model by a KL penalty. People preferred its answers strongly, it made up facts less often, and the recipe became the basis of ChatGPT-style assistants.

What you will learn

  • What is the InstructGPT paper?
  • The building blocks we must know first
  • The big picture: what InstructGPT does
  • Why GPT-3 was not enough
  • Helpful, Honest, and Harmless
  • The three-step method
  • Step 1: Supervised Fine-Tuning
  • Step 2: The Reward Model
  • Step 3: Reinforcement Learning with PPO
  • The alignment tax
  • The results
  • What alignment looks like today
  • Quick Summary

In this lesson

  1. What is the InstructGPT paper?
  2. The building blocks we must know first
  3. Why GPT-3 was not enough
  4. Helpful, honest, and harmless
  5. The three-step method
  6. Step 1: supervised fine-tuning
  7. Step 2: the reward model
  8. Step 3: reinforcement learning with PPO
  9. The alignment tax and the results
  10. Going one level deeper
  11. Practice: try it yourself
  12. What alignment looks like today, and a quick summary

Key terms in this lesson

InstructGPT
GPT-3 models fine-tuned with SFT and RLHF to follow instructions (OpenAI, 2022).
Alignment
Making a model’s behaviour match what its users and developers intend.
SFT
Supervised fine-tuning on human-written demonstration answers.
Reward model
A model that outputs a scalar score predicting how much humans would like a response.
KL penalty
A cost for the policy’s distribution moving away from the reference (SFT) model.
Alignment tax
Performance lost on some tasks as a side effect of alignment training.
PPO-ptx
PPO training mixed with pretraining-data gradients to limit the alignment tax.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 8.6 Deep RL from Human Preferences: The Foundational Paper · 8.8 RLHF: Aligning LLMs with Human Preferences →