Modern AI Engineering

Lesson 8.6 · 26 min · video lesson

Deep RL from Human Preferences: The Foundational Paper

How do you reward a robot for doing a graceful backflip when nobody can write down, in code, what “graceful backflip” means?

In short: The 2017 paper “Deep Reinforcement Learning from Human Preferences” showed that an agent can learn complex behaviour without a hand-written reward. Humans watch pairs of short clips and pick the better one; a reward model learns to predict those choices; and a reinforcement learning agent maximises the learned reward. With feedback on under 1% of the agent’s interactions it learned Atari games, robot locomotion and even a backflip. This loop is the foundation of RLHF.

What you will learn

  • The building blocks we must know first
  • The big picture: what the paper does
  • Why it was needed: the reward problem
  • Trajectory segments, or clips
  • The human comparison
  • The reward model and the preference math
  • Training the reward model
  • Training the agent with reinforcement learning
  • The loop and the smart bits
  • The results: the backflip and beyond
  • The legacy: this is RLHF
  • What this looks like today
  • Quick Summary

In this lesson

  1. The building blocks we must know first
  2. Why it was needed: the reward problem
  3. The big picture: what the paper does
  4. Trajectory segments and the human comparison
  5. The reward model and the preference math
  6. Training the reward model and the agent
  7. The results: the backflip and beyond
  8. The legacy: this is RLHF
  9. Worked example, step by step
  10. Practice: try it yourself
  11. What this looks like today, and a quick summary

Key terms in this lesson

Reinforcement learning
Learning a policy by trial and error to maximise total reward.
Policy
The agent’s rule for choosing an action in each state.
Trajectory segment (clip)
A short piece of the agent’s behaviour, shown to humans for comparison.
Reward model
A network trained to predict rewards that agree with human preferences.
Bradley–Terry model
A model where the chance one item beats another is the sigmoid of their score difference.
Reward hacking
An agent scoring high on a flawed reward without doing what was intended.
Active learning
Choosing which examples to label so each label teaches the model the most.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 8.5 Continual Learning: Training Without Forgetting the Past · 8.7 InstructGPT: Teaching GPT-3 to Follow Instructions →