Modern AI Engineering

Lesson 13.17 · 31 min · video lesson

TensorRT-LLM: NVIDIA's Optimized Inference Engine

If a GPU can do trillions of operations per second, why does it often sit idle while serving an LLM, and how does NVIDIA's TensorRT-LLM get that time back?

In short: TensorRT-LLM is NVIDIA's open-source library for running LLMs as fast as possible on NVIDIA GPUs. It prepares the model ahead of time (fusing operations into fewer kernels, picking the fastest kernel for each operation, and using low-precision formats like FP8), then serves it with a paged KV cache, in-flight batching, CUDA graphs, speculative decoding and multi-GPU parallelism. Newer versions add a PyTorch backend that keeps most of the speed without a separate build step.

What you will learn

  • What is inference
  • What is a GPU and what is a kernel
  • The problem: the GPU spends its time on the wrong things
  • What is TensorRT-LLM
  • The big idea: prepare the model ahead of time
  • The build step: from a model to an engine
  • Kernel fusion
  • Quantization
  • Custom attention kernels
  • The paged KV cache
  • In-flight batching
  • CUDA graphs
  • Speculative decoding
  • Running one model across many GPUs
  • How we actually serve the model
  • The PyTorch backend, the newer and easier path
  • The full journey of one request
  • TensorRT-LLM vs vLLM
  • Where it works well and where it fails
  • [LLM Inference Optimization](https://www.youtube.com/watch?v=jV2sCj4lHYk) (Video)
  • [The First-Token Latency Problem in LLMs](https://www.youtube.com/watch?v=XD8DD4cEHu0) (Video)
  • The full LLM inference optimization pipeline, end to end

In this lesson

  1. What is inference
  2. What is a GPU and what is a kernel
  3. The problem: the GPU spends its time on the wrong things
  4. What is TensorRT-LLM, and the big idea
  5. The build step: from a model to an engine
  6. Kernel fusion
  7. Quantization and custom attention kernels
  8. The paged KV cache, in-flight batching and CUDA graphs
  9. Speculative decoding and running one model across many GPUs
  10. How we serve the model, and the PyTorch backend
  11. TensorRT-LLM vs vLLM, and where it works well or fails
  12. Going one level deeper
  13. Practice: try it yourself

Key terms in this lesson

Kernel
A single function that runs on the GPU, such as a matrix multiply or a fused activation.
Engine
TensorRT-LLM's compiled, optimized execution plan for one model, GPU architecture, precision and set of limits.
Kernel fusion
Combining several operations into one kernel to cut memory traffic and launch overhead.
In-flight batching
NVIDIA's term for continuous batching: re-forming the batch at every iteration.
CUDA graph
A recorded sequence of GPU kernel launches that can be replayed with a single launch.
Tensor parallelism
Splitting each weight matrix across GPUs so they compute every layer together.
FP8
An 8-bit floating-point format that recent NVIDIA GPUs can multiply natively and fast.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 13.16 SGLang: Structured LLM Programs for Efficient Inference · 14.1 Evaluating LLMs: Metrics, Benchmarks, and Methods →