Modern AI Engineering

Lesson 18.1 · 29 min · video lesson

JEPA: LeCun's Vision for World Model AI

If you see half of a dog behind a fence, you instantly “know” the rest is there, without imagining every hair. Can a machine learn to predict like that?

In short: JEPA (Joint Embedding Predictive Architecture) is a self-supervised learning design proposed by Yann LeCun. Instead of predicting the missing part of an input pixel by pixel, it encodes both the visible part and the hidden part into embeddings and trains a predictor to guess the hidden part's embedding from the visible part's. This lets the model ignore unpredictable detail and focus on meaning, avoids the need for negative examples, and is a key building block in LeCun's vision of world models. I-JEPA applies it to images and V-JEPA to video.

What you will learn

  • How humans and animals learn by observing the world
  • Yann LeCun's vision of autonomous machine intelligence
  • A simple everyday analogy to build the intuition
  • What does JEPA mean
  • What is an embedding or representation space
  • The problem with predicting raw pixels
  • The problem with contrastive methods
  • The core idea of JEPA
  • The building blocks of JEPA
  • The energy-based view in simple words
  • How I-JEPA works (for images)
  • V-JEPA and the world-model vision
  • When and why JEPA matters

In this lesson

  1. Learning by observing the world
  2. What does JEPA mean? Embeddings and representation space
  3. The problems with predicting pixels and with contrastive methods
  4. The core idea and building blocks of JEPA
  5. The energy-based view in simple words
  6. Seeing the idea in numbers
  7. How I-JEPA works (for images)
  8. V-JEPA and the world-model vision
  9. Comparison, and when and why JEPA matters
  10. Going one level deeper
  11. Practice: try it yourself

Key terms in this lesson

Self-supervised learning
Learning from unlabelled data by predicting hidden parts of the data from other parts.
Representation (embedding)
A vector of numbers summarising an input so that similar content gives similar vectors.
JEPA
Joint Embedding Predictive Architecture: predicts the embedding of a hidden part of the input from the embedding of the visible part.
Representation collapse
A failure where encoders map every input to (nearly) the same vector, making the loss trivially small.
EMA target encoder
An encoder whose weights are an exponential moving average of the trained encoder's weights, giving stable targets.
Energy-based model
A model that scores compatibility of input pairs with an energy: low for compatible, high for incompatible.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 17.11 How do Voice And Video Call Work? · 18.2 World Models: Teaching AI to Simulate Its Environment →