Modern AI Engineering

Lesson 16.2 · 25 min · video lesson

Vision Transformers: Applying Self-Attention to Image Patches

Transformers were built to read sentences — so how can the very same architecture look at a photo and say 'that is a cracked blender jar'?

In short: A Vision Transformer (ViT) treats an image like a sentence: it cuts the image into small square patches, turns each patch into a vector (a 'visual word'), adds a special [CLS] token and position information, and runs the sequence through a standard Transformer encoder. The final [CLS] vector summarises the whole image and a small head turns it into class scores. ViTs need more data than CNNs to learn well, but they scale very well and are now the default image encoder inside most multimodal models.

What you will learn

  • The Big Picture
  • Decoding Step 1: Splitting the Image into Patches
  • Decoding Step 2: Patch Embedding
  • Decoding Step 3: The CLS Token
  • Decoding Step 4: Position Embeddings
  • Decoding Step 5: The Transformer Encoder
  • Decoding Step 6: The Classification Head
  • Putting It All Together
  • ViT vs CNN
  • Quick Summary

In this lesson

  1. The big picture
  2. Decoding step 1: splitting the image into patches
  3. Decoding step 2: patch embedding
  4. Decoding step 3: the [CLS] token
  5. Decoding step 4: position embeddings
  6. Decoding step 5: the Transformer encoder
  7. Decoding step 6: the classification head
  8. Putting it all together (with code)
  9. ViT vs CNN
  10. Worked example, step by step
  11. Practice: try it yourself
  12. Quick summary

Key terms in this lesson

Patch
A small non-overlapping square of the image (e.g. 16×16 pixels) that becomes one token.
Patch embedding
The vector produced by projecting a flattened patch with a shared learned matrix.
[CLS] token
An extra learnable token placed first in the sequence whose final output summarises the image.
Position embedding
A learned vector added to each token so the model knows which position it came from.
Inductive bias
An assumption built into a model's design, such as locality in CNNs.
Receptive field
The region of the input that can influence a given feature.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 16.1 Multimodal AI: Perceiving Text, Images, and Audio Together · 16.3 Image Embeddings: Encoding Visual Content as Vectors →