Modern AI Engineering

Lesson 16.6 · 26 min · video lesson

Variational Autoencoders: Learning a Compressed Latent Space

A normal autoencoder can squeeze a photo into a few numbers and rebuild it — so why can it not invent a new photo when we hand it a few random numbers?

In short: An autoencoder compresses data into a small latent code and reconstructs it, but its latent space has gaps, so random codes decode to garbage. A Variational Autoencoder (VAE) fixes this by making the encoder output a small probability cloud (a mean and a spread) instead of a single point, and by adding a KL penalty that pulls all clouds toward a standard normal distribution. The result is a smooth, well-organised latent space we can sample from to generate new data; the reparameterization trick makes this trainable with backpropagation.

What you will learn

  • What is an Autoencoder?
  • The problem with a normal Autoencoder
  • What is a Variational Autoencoder?
  • The encoder, the latent space, and the decoder
  • The reparameterization trick
  • The loss function of a Variational Autoencoder
  • A simple example walk-through
  • A simple code example
  • Advantages of Variational Autoencoders
  • Where Variational Autoencoders are used

In this lesson

  1. What is an autoencoder?
  2. The problem with a normal autoencoder
  3. What is a Variational Autoencoder?
  4. The encoder, the latent space, and the decoder
  5. The reparameterization trick
  6. The loss function of a Variational Autoencoder
  7. A simple example walk-through, in code
  8. Advantages of Variational Autoencoders
  9. Where Variational Autoencoders are used
  10. Common mistakes and how to spot them
  11. Practice: try it yourself

Key terms in this lesson

Autoencoder
A network that compresses input to a latent code and reconstructs it.
Latent space
The space of codes z produced by the encoder and consumed by the decoder.
Prior
The distribution we want latent codes to follow, usually the standard normal N(0, I).
KL divergence
A measure of how different one probability distribution is from another; zero when they are equal.
Reparameterization trick
Writing z = μ + σ ⊙ ε with ε ~ N(0, I) so that sampling becomes differentiable.
ELBO
Evidence Lower Bound: the training objective of a VAE, a lower bound on the log-likelihood.
Posterior collapse
A failure where the decoder ignores z and the encoder's output equals the prior.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 16.5 GANs: A Generator and Discriminator in Constant Competition · 17.1 GPUs for Deep Learning: Parallelism at the Core →