Modern AI Engineering

Lesson 10.9 · 22 min · video lesson

Embedding Caches: Avoiding Redundant Embedding Calls

Every night our RAG pipeline re-embeds 2 million chunks, and 98% of them have not changed since yesterday. Why are we paying for the same numbers again?

In short: An embedding cache stores the vector computed for a piece of text so the same text never has to be embedded twice. The cache key is a hash of the exact text together with the model name (and version), the value is the vector, and a lookup either hits (return the stored vector) or misses (call the model, store the result). Eviction rules such as LRU and TTL keep it bounded, and it can live in memory, on disk or in a shared store like Redis.

What you will learn

  • What is an embedding
  • A quick recap of how we get an embedding
  • What is an Embedding Cache
  • Why we need an Embedding Cache
  • The core idea behind an Embedding Cache
  • The cache key, a hash of the text plus the model
  • The request flow, a hit and a miss
  • Eviction, LRU and TTL
  • Where the cache lives, memory or disk
  • The benefits of an Embedding Cache
  • An Embedding Cache in the real world

In this lesson

  1. What is an embedding, and how do we get one?
  2. What is an embedding cache, and why do we need one?
  3. The core idea and the cache key: hash of text plus model
  4. The request flow: a hit and a miss
  5. Eviction: LRU and TTL
  6. Where the cache lives: memory or disk
  7. Benefits and the embedding cache in the real world
  8. Worked example, step by step
  9. Practice: try it yourself

Key terms in this lesson

Embedding cache
A key-value store that maps (model, text) to its previously computed vector.
Cache hit / miss
Whether the requested key is already in the cache (hit) or must be computed (miss).
Hash (SHA-256)
A function that turns any input into a fixed-length fingerprint that changes completely if the input changes.
LRU
Least Recently Used eviction: remove the entry unused for the longest time.
TTL
Time To Live: an entry expires a fixed time after it was stored.
Cache namespace
A separate key space, e.g. one per embedding model, so entries never mix.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 10.8 HyDE: Generating Hypothetical Documents to Improve RAG · 10.10 Semantic Caching: Skipping the LLM for Similar Queries →