Modern AI Engineering

Lesson 4.4 · 27 min · video lesson

Embeddings: Encoding Meaning as Vectors

How can a computer know that “puppy” is closer to “dog” than to “invoice” when, to it, every word is just a string of characters?

In short: An embedding is a list of numbers (a vector) that represents a piece of data, such as a word, a sentence, an image or a product, so that similar things get similar numbers. We measure how close two embeddings are with cosine similarity or distance, which lets computers compare meaning. Embeddings are learned by neural networks from data, power search, RAG, recommendations and clustering, and come with real pitfalls: bias, model mismatch and domain gaps.

What you will learn

  • The problem: computers cannot compare meaning
  • What are Embeddings?
  • A simple example with two numbers
  • Why we need many more than two numbers
  • How do we measure closeness?
  • Where do embeddings come from?
  • The famous word math example
  • Embeddings are not only for words
  • What we use embeddings for
  • Things we must be careful about
  • Summary

In this lesson

  1. The problem: computers cannot compare meaning
  2. What are Embeddings?
  3. Why we need many more than two numbers
  4. How do we measure closeness?
  5. Where do embeddings come from?
  6. The famous word math example
  7. Embeddings are not only for words
  8. What we use embeddings for
  9. Things we must be careful about
  10. Worked example, step by step
  11. Practice: try it yourself
  12. Summary

Key terms in this lesson

Embedding
A dense vector of numbers representing an item so that similar items are close together.
Dimension
One number in a vector; the dimensionality is how many numbers the vector has.
One-hot vector
A vector with a single 1 marking a category and 0s elsewhere; it has no notion of similarity.
Cosine similarity
The cosine of the angle between two vectors, from −1 to 1, ignoring their lengths.
Contextual embedding
A vector for a token or text computed from its surrounding context, typically by a Transformer.
Distributional hypothesis
The idea that words appearing in similar contexts have similar meanings.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 4.3 BPE Tokenization: How LLMs Split Text into Tokens · 4.5 RNNs vs Transformers: A Fundamental Architecture Shift →