Modern AI Engineering

Lesson 9.3 · 22 min · video lesson

Prompt Caching: Reusing Computation Across API Calls

If our chatbot sends the same 20,000-token manual with every question, why should the model read it from scratch every single time?

In short: When an LLM reads a prompt it computes internal key and value tensors for every token. Prompt caching stores those tensors for a prompt prefix so the next request that starts with exactly the same tokens can skip that work. The result is lower latency and much cheaper input tokens, as long as we put the stable content first and keep it byte-for-byte identical.

What you will learn

  • What is a prompt
  • A quick recap of how an LLM reads a prompt
  • What is Prompt Caching
  • Why we need Prompt Caching
  • The core idea behind Prompt Caching
  • The exact-prefix rule
  • Cache write vs cache read and TTL
  • What we should put in the cache
  • The benefits of Prompt Caching
  • Prompt Caching in the real world

In this lesson

  1. What is a prompt, and how does an LLM read it?
  2. What is prompt caching, and why do we need it?
  3. The core idea: a prefix's keys and values never change
  4. The exact-prefix rule
  5. Cache write vs cache read, and TTL
  6. Code you can run: a toy prefix cache with prices
  7. What we should put in the cache
  8. Benefits, real-world use and pitfalls
  9. Worked example, step by step
  10. Practice: try it yourself

Key terms in this lesson

Prefill
The phase where the model processes all prompt tokens and builds the KV cache.
KV cache
Stored key and value vectors for processed tokens, reused during generation.
Prompt caching
Keeping the KV cache of a prompt prefix so later requests with the same prefix skip its prefill.
Exact-prefix rule
A cache hit requires the prompt to match token-for-token from the start up to the cached point.
Cache write / cache read
Storing a new prefix on a miss vs reusing a stored prefix on a hit; often priced differently.
TTL
Time to live: how long an unused cache entry is kept before it expires.
Time to first token (TTFT)
Delay between sending a request and receiving the first generated token.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 9.2 Prompt Chaining: Decomposing Complex Tasks into Steps · 9.4 Context Engineering: Curating the Model's Working Memory →