Modern AI Engineering

Lesson 10.10 · 23 min · video lesson

Semantic Caching: Skipping the LLM for Similar Queries

"How do I get a refund?" and "Can I get my money back?" are different strings but the same question. Why pay an LLM twice to answer it?

In short: A semantic cache stores past LLM answers together with the embedding of the question that produced them. When a new question arrives, we embed it and look for a stored question whose embedding is similar enough (above a threshold); if found, we return the stored answer instead of calling the LLM. It saves cost and latency on repeated questions, but the threshold must be tuned carefully, because a too-low threshold returns answers to questions that only look similar.

What you will learn

  • What is a cache?
  • The problem with traditional caching for AI apps
  • What is Semantic Caching?
  • What are embeddings?
  • What is similarity between embeddings?
  • How does Semantic Caching work step by step?
  • A numeric walkthrough
  • Setting the similarity threshold
  • Advantages of Semantic Caching
  • Things to keep in mind

In this lesson

  1. What is a cache, and why traditional caching fails for AI apps
  2. What is semantic caching? Embeddings and similarity
  3. How semantic caching works step by step
  4. A numeric walkthrough in code
  5. Setting the similarity threshold
  6. Advantages of semantic caching
  7. Things to keep in mind
  8. Worked example, step by step
  9. Practice: try it yourself

Key terms in this lesson

Semantic cache
A cache that reuses stored LLM answers for new questions with similar meaning.
Similarity threshold
The minimum similarity between question embeddings that counts as a cache hit.
False hit
A cache hit that returns the answer to a different question.
Cosine similarity
The cosine of the angle between two vectors; close to 1 means very similar meaning.
TTL
Time to live: how long a cache entry stays valid before it expires.
Prompt caching
A provider feature that reuses computation for identical prompt prefixes; it still generates a fresh answer.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 10.9 Embedding Caches: Avoiding Redundant Embedding Calls · 10.11 Agentic RAG: Dynamic Retrieval with Multi-Step Reasoning →