Modern AI Engineering

Lesson 5.4 · 23 min · video lesson

Lost in the Middle: Why LLMs Miss Central Context

You give a model 20 documents and the answer is right there in document 10. Why might it do worse than if you had given it no documents at all?

In short: LLMs tend to use information at the beginning and end of a long context much better than information in the middle. Plotting accuracy against the position of the key fact gives a U-shaped curve. This "lost in the middle" effect, documented in a 2023 study, matters for RAG, long chats and document analysis. We can test for it by moving a known fact through the context, and reduce it by sending fewer, better-ranked chunks, placing the most relevant ones at the edges, and putting the question after the documents.

What you will learn

  • What is a context window
  • What is the Lost in the Middle problem
  • Let's understand it with an example
  • The U-shaped curve
  • Why does this happen
  • Where this hurts us in real life
  • How to test for the Lost in the Middle problem
  • How to solve the Lost in the Middle problem
  • Key points to remember
  • [Why is the context window limited in LLMs?](https://www.youtube.com/watch?v=CGIhxIaOg3M) (Video)

In this lesson

  1. What is a context window?
  2. What is the lost-in-the-middle problem?
  3. Let us understand it with an example
  4. The U-shaped curve
  5. Why does this happen?
  6. Where this hurts us in real life
  7. How to test for the lost-in-the-middle problem
  8. How to solve the lost-in-the-middle problem
  9. Worked example, step by step
  10. Practice: try it yourself
  11. Key points to remember

Key terms in this lesson

Context window
The maximum number of tokens a model can take into account at once.
Lost in the middle
The tendency to under-use information placed in the middle of a long context.
Primacy effect
Better use or recall of information that appears first.
Recency effect
Better use or recall of information that appears last.
Needle in a haystack test
Hiding a known fact at different depths of filler text and checking if the model finds it.
Reranker
A model that re-scores retrieved chunks so only the most relevant few are sent to the LLM.
Map-reduce
Processing chunks separately and then combining the partial results.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 5.3 Token Streaming: Rendering Outputs as They Arrive · 6.1 A Timeline of LLM Architecture Improvements →