Modern AI Engineering

Lesson 10.6 · 29 min · video lesson

ColBERT: Token-Level Late Interaction for Retrieval

What if, instead of squeezing a whole passage into one vector, we kept a vector for every word and let each query word find its best match?

In short: ColBERT (Khattab and Zaharia, 2020) is a retrieval model that encodes the query and each document separately into one vector per token, then scores a pair with MaxSim: every query token takes its highest similarity to any document token, and these maxima are summed. This "late interaction" keeps most of the precision of a cross-encoder while letting document vectors be computed ahead of time, at the cost of a much bigger index.

What you will learn

  • What is the ColBERT paper?
  • The building blocks we must know first
  • The big picture: what ColBERT does
  • The two old extremes
  • Late interaction: the key idea
  • Encoding the query and document
  • The MaxSim operation
  • Why max, not average
  • Ranking documents
  • Training: positives and negatives
  • The loss with small numbers
  • Fast retrieval at scale
  • The cost of a bigger index
  • The Results
  • Where ColBERT led
  • Quick Summary

In this lesson

  1. What is the ColBERT paper, and what we need to know first
  2. The big picture: the two old extremes
  3. Late interaction and how the query and document are encoded
  4. The MaxSim operation, and why max, not average
  5. Ranking documents, with code
  6. Training with positives and negatives: the loss with small numbers
  7. Fast retrieval at scale, and the cost of a bigger index
  8. The results, and where ColBERT led
  9. Going one level deeper
  10. Practice: try it yourself
  11. Quick summary

Key terms in this lesson

ColBERT
Contextualized Late interaction over BERT: a multi-vector retrieval model scored with MaxSim.
Late interaction
Encoding texts separately and letting their token vectors interact only at scoring time.
MaxSim
For each query token, the highest similarity to any document token; summed over the query.
Contextualized embedding
A token vector that depends on the surrounding words.
Hard negative
A non-relevant passage that looks relevant, used to sharpen training.
Query augmentation
Padding the query with [MASK] tokens whose vectors act as learned query expansions.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 10.5 Rerankers: Re-Scoring Retrieved Results by Relevance · 10.7 Document Chunking Strategies for RAG →