Lesson 10.7 · 31 min · video lesson
Document Chunking Strategies for RAG
Your RAG bot has the right document, yet it answers "I don't know". Very often the culprit is not the model or the search, but where we cut the text.
In short: RAG retrieves pieces of documents, called chunks, and hands them to an LLM. How we cut those chunks decides what can be found and whether the retrieved text makes sense on its own. Strategies range from simple fixed-size windows and sentence packing to recursive, structure-aware, semantic, contextual, small-to-big and LLM-driven (agentic) chunking; the right choice depends on the documents, the questions and the budget.
What you will learn
- What is RAG?
- What is a chunk?
- Why do we need chunking?
- How retrieval actually works
- What happens when we chunk badly
- Fixed-size chunking
- Chunking by sentence
- Recursive chunking
- Document structure based chunking
- Semantic chunking
- Contextual chunking
- Small-to-big chunking
- Agentic chunking
- Chunk overlap
- How to choose the chunk size
- Comparison of all the strategies
- Common mistakes
- Conclusion
In this lesson
- What is RAG, what is a chunk, and why chunk at all?
- How retrieval actually works, and what bad chunking breaks
- Fixed-size, sentence and recursive chunking
- Document-structure and semantic chunking
- Contextual, small-to-big and agentic chunking
- Chunk overlap, and how to choose the chunk size
- Comparison of all the strategies
- Worked example, step by step
- Practice: try it yourself
- Common mistakes and conclusion
Key terms in this lesson
- Chunk
- A piece of a document stored, embedded and retrieved as one unit.
- Chunk overlap
- Text repeated at the boundary between neighbouring chunks.
- Recursive chunking
- Splitting by a hierarchy of separators (paragraph, sentence, word) until pieces fit a size limit.
- Semantic chunking
- Cutting where embedding similarity between neighbouring sentences drops.
- Contextual chunking
- Adding document-level context to each chunk before embedding it.
- Small-to-big
- Searching small child chunks but returning their larger parent chunks to the LLM.
- Proposition
- A short, self-contained statement of a single fact, used in agentic chunking.
This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).
← 10.6 ColBERT: Token-Level Late Interaction for Retrieval · 10.8 HyDE: Generating Hypothetical Documents to Improve RAG →