Modern AI Engineering

Lesson 13.14 · 27 min · video lesson

llama.cpp: Running Large Models on Consumer Hardware

How does a model that was trained on thousands of data-centre GPUs end up answering questions on an ordinary laptop with no internet connection?

In short: llama.cpp is an open-source C/C++ engine that runs LLMs on everyday CPUs and GPUs. It combines four ideas: quantize the weights to about 4–8 bits so they fit, pack everything into one GGUF file, memory-map that file so it loads instantly, and squeeze speed out of the hardware with SIMD instructions, threads and optional GPU offload. Because token generation is limited by memory bandwidth, smaller weights translate almost directly into faster answers.

What you will learn

  • What is llama.cpp
  • Why we needed llama.cpp
  • A quick refresher on what an LLM is
  • The real problem: models are too big to fit
  • The first big idea: quantization
  • Understanding names like Q4_K_M
  • The GGUF file: everything packed in one box
  • Memory mapping: loading the model the smart way
  • Squeezing speed out of the CPU
  • Sharing the work with the GPU
  • The full journey of running a prompt
  • Where llama.cpp is used

In this lesson

  1. What is llama.cpp
  2. Why we needed llama.cpp
  3. A quick refresher on what an LLM is
  4. The real problem: models are too big to fit
  5. The first big idea: quantization, and names like Q4_K_M
  6. The GGUF file and memory mapping
  7. Squeezing speed out of the CPU
  8. Sharing the work with the GPU
  9. The full journey of running a prompt
  10. Where llama.cpp is used
  11. Worked example, step by step
  12. Practice: try it yourself

Key terms in this lesson

llama.cpp
An open-source C/C++ engine for running LLMs efficiently on CPUs, Apple silicon and consumer GPUs.
ggml
The tensor library underneath llama.cpp that provides quantized maths and hardware backends.
SIMD
Single Instruction, Multiple Data: CPU instructions (AVX, NEON) that process many numbers at once.
Layer offload (-ngl)
Placing the weights of some transformer layers on the GPU while the rest run on the CPU.
Memory bandwidth
How many bytes per second a processor can read from memory; the main limit on decode speed.
Unified memory
A single memory pool shared by CPU and GPU, as on Apple silicon, so the GPU can use most of system RAM.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 13.13 GGUF: The File Format Powering Local LLM Inference · 13.15 vLLM: High-Throughput Serving with PagedAttention →