Lesson 13.12 · 27 min · video lesson

Model Quantization: Shrinking Weights Without Breaking Outputs

How can a 7-billion-parameter model that needs 28 GB in full precision squeeze into 4 GB and still give nearly the same answers?

In short: Quantization stores a model's numbers with fewer bits, for example 8-bit or 4-bit integers instead of 32-bit floats, using a scale (and sometimes a zero-point) to map between the two. That cuts memory by 4–8× and speeds up the memory-bound parts of inference. The art is choosing the right granularity, method and bit width so that rare large values (outliers) do not destroy accuracy.

What you will learn

What is Model Quantization?
How numbers use bits (FP32, INT8, INT4)
Why fewer bits means less memory and faster speed
The core mechanic: scale and zero-point
Symmetric vs Asymmetric Quantization
Per-tensor vs Per-channel Quantization
Post-Training Quantization (PTQ) vs Quantization-Aware Training (QAT)
Weight-only vs Weight-and-activation Quantization
The outlier problem in LLMs
Popular methods: GPTQ, AWQ, bitsandbytes, GGUF / llama.cpp
The accuracy trade-off and running LLMs locally
Wrapping up Model Quantization

In this lesson

What is model quantization?
How numbers use bits (FP32, INT8, INT4)
Why fewer bits means less memory and faster speed
The core mechanic: scale and zero-point
Symmetric vs asymmetric quantization
Per-tensor vs per-channel quantization
PTQ vs QAT, and weight-only vs weight-and-activation
The outlier problem in LLMs
Popular methods: GPTQ, AWQ, bitsandbytes, GGUF / llama.cpp
The accuracy trade-off and running LLMs locally
Worked example, step by step
Practice: try it yourself
Wrapping up model quantization

Key terms in this lesson

Quantization: Representing numbers with fewer bits by rounding them to a coarse grid of allowed values.
Scale: The real-valued size of one integer step; multiply an integer by it to get back an approximate real value.
Zero-point: The integer that represents real 0.0 in asymmetric quantization.
Per-channel / per-group: Using a separate scale for each row or each small block of weights instead of one per tensor.
PTQ: Post-Training Quantization: converting an already-trained model to low precision, often with a small calibration set.
QAT: Quantization-Aware Training: simulating rounding during training so the model learns to tolerate it.
Outlier features: A few activation dimensions in large LLMs with values far bigger than the rest, which break naive quantization.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 13.11 EAGLE: Feature-Level Drafting for Faster Inference · 13.13 GGUF: The File Format Powering Local LLM Inference →