Modern AI Engineering

Lesson 17.2 · 28 min · video lesson

CUDA Kernels: Writing Parallel Code for NVIDIA GPUs

When PyTorch multiplies two matrices on a GPU, a million tiny threads each run the same short function — so how does each one know which numbers are its job?

In short: A CUDA kernel is a function written once and executed in parallel by thousands or millions of GPU threads. Threads are grouped into blocks, and blocks into a grid; each thread computes its own global index from built-in variables and works on the matching piece of data. The CPU (host) copies data to GPU memory and launches kernels; the GPU (device) schedules blocks onto its streaming multiprocessors in groups of 32 threads called warps. Fast kernels depend as much on smart memory use as on raw arithmetic.

What you will learn

  • Why do we need a GPU?
  • What is CUDA?
  • What is a CUDA Kernel?
  • Threads, Blocks, and Grids
  • Host and Device
  • Writing our first CUDA Kernel
  • How a thread finds its own work
  • What happens inside the GPU when a kernel runs
  • Memory in CUDA
  • Why CUDA Kernels matter for AI
  • Where CUDA Kernels work well and where they fail

In this lesson

  1. Why do we need a GPU?
  2. What is CUDA?
  3. What is a CUDA kernel?
  4. Threads, blocks, and grids
  5. Host and device
  6. Writing our first CUDA kernel
  7. How a thread finds its own work
  8. What happens inside the GPU when a kernel runs
  9. Memory in CUDA
  10. Why CUDA kernels matter for AI
  11. Where CUDA kernels work well and where they fail
  12. Worked example, step by step
  13. Practice: try it yourself

Key terms in this lesson

CUDA
NVIDIA's platform and C++ extension for general-purpose GPU programming.
Kernel
A function that runs on the GPU, executed by many threads in parallel.
Thread block
A group of up to 1,024 threads that run on one SM and can share memory and synchronise.
Grid
All the thread blocks of one kernel launch.
Warp
A group of 32 threads that execute the same instruction together.
Streaming Multiprocessor (SM)
One of the GPU's processing units that runs blocks of threads.
Coalesced access
Neighbouring threads reading neighbouring addresses so the hardware can combine their memory requests.
Kernel fusion
Combining several operations into one kernel to avoid round trips to global memory.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 17.1 GPUs for Deep Learning: Parallelism at the Core · 17.3 Google TPUs: Purpose-Built Hardware for Neural Networks →