Modern AI Engineering

Lesson 4.3 · 25 min · video lesson

BPE Tokenization: How LLMs Split Text into Tokens

How can a model with a fixed list of about 100,000 tokens read any word ever written, including typos, names and emoji it has never seen?

In short: Before a language model can read text, a tokenizer cuts the text into tokens and maps each one to an integer ID. Byte Pair Encoding (BPE) builds that token list by starting from single characters (or bytes) and repeatedly merging the most frequent neighbouring pair, so common words become one token while rare words split into reusable pieces. To tokenize new text it replays the learned merges in order, which is why BPE never meets a truly unknown word.

What you will learn

  • What is Tokenization?
  • The Problem: How to Break Text into Tokens?
  • What is BPE (Byte Pair Encoding)?
  • How BPE Works: Step by Step
  • How BPE Tokenizes New Text
  • Why BPE is Used in Modern LLMs

In this lesson

  1. What is Tokenization?
  2. The Problem: How to Break Text into Tokens?
  3. What is BPE (Byte Pair Encoding)?
  4. How BPE Works: Step by Step
  5. Code: train BPE and tokenize new words
  6. How BPE Tokenizes New Text
  7. Why BPE is Used in Modern LLMs
  8. Common mistakes and limits
  9. Going one level deeper
  10. Practice: try it yourself

Key terms in this lesson

Token
A unit of text (word, word piece, character or byte) that the model reads as one ID.
Vocabulary
The fixed list of all tokens a tokenizer can produce, each with an integer ID.
Byte Pair Encoding (BPE)
A subword method that builds a vocabulary by repeatedly merging the most frequent adjacent pair of symbols.
Merge rule
One learned instruction to glue two adjacent symbols into a new symbol, applied in learned order.
Byte-level BPE
BPE that starts from the 256 byte values, so any text can be encoded.
Out-of-vocabulary (OOV)
A word a tokenizer cannot represent; byte-level BPE avoids this.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 4.2 Autoregressive Models: Predicting One Token at a Time · 4.4 Embeddings: Encoding Meaning as Vectors →