Modern AI Engineering

Lesson 13.13 · 23 min · video lesson

GGUF: The File Format Powering Local LLM Inference

Why can you download one file called something like model-Q4_K_M.gguf, double-click it in a desktop app, and be chatting with an LLM a few seconds later?

In short: GGUF is a single-file format for storing an LLM: a small header, a dictionary of metadata (architecture, tokenizer, chat template), a table describing each tensor, and then the (usually quantized) weights, aligned so they can be memory-mapped. Names like Q4_K_M describe how the weights were quantized. Because everything is in one self-describing file that loads almost instantly, GGUF became the standard for running models locally with llama.cpp, Ollama and LM Studio.

What you will learn

  • What is a model and what are weights
  • What is local inference
  • The problem before GGUF
  • What is GGUF
  • What is stored inside a GGUF file
  • What is quantization
  • Understanding quantization names like Q4_K_M
  • How GGUF loads fast with memory mapping
  • Why GGUF is cross-platform and extensible
  • GGUF in the real world

In this lesson

  1. What is a model and what are weights
  2. What is local inference
  3. The problem before GGUF
  4. What is GGUF
  5. What is stored inside a GGUF file
  6. What is quantization (in GGUF)
  7. Understanding quantization names like Q4_K_M
  8. How GGUF loads fast with memory mapping
  9. Why GGUF is cross-platform and extensible
  10. GGUF in the real world
  11. Worked example, step by step
  12. Practice: try it yourself

Key terms in this lesson

GGUF
The single-file binary format from the ggml / llama.cpp project for storing models with metadata and quantized tensors.
Tensor
A named multi-dimensional array of numbers, such as one weight matrix of the model.
Metadata
Named, typed key–value pairs in the file that describe the model, its settings and its tokenizer.
Block quantization
Quantizing weights in small groups, each with its own scale, so one large value affects only its block.
k-quant
GGUF quantization types (Q2_K … Q6_K) using 256-weight super-blocks with quantized sub-block scales.
Memory mapping (mmap)
An OS feature that makes a file appear in memory and loads its pages from disk only when they are accessed.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 13.12 Model Quantization: Shrinking Weights Without Breaking Outputs · 13.14 llama.cpp: Running Large Models on Consumer Hardware →