Modern AI Engineering

Lesson 6.7 · 31 min · video lesson

DeepSeek-V4: Anatomy of an Open-Source Frontier Model

How does an open model read a million tokens while keeping only about a tenth of the memory its predecessor needed?

In short: DeepSeek-V4 (April 2026) is a pair of open Mixture-of-Experts models, V4-Pro (1.6T total, 49B active parameters) and V4-Flash (284B total, 13B active), both with a native 1M-token context. Its main ideas are a hybrid attention that compresses past tokens (CSA: compress 4× and pick the top-k blocks; HCA: compress 128× and attend to all), stronger residual connections called mHC, the Muon optimizer, FP4 quantization-aware training, and a post-training recipe that trains domain specialists and merges them by on-policy distillation. Users choose among three reasoning modes.

What you will learn

  • The Big Picture
  • Two Models: DeepSeek-V4-Pro and DeepSeek-V4-Flash
  • Hybrid Attention with CSA and HCA
  • Manifold-Constrained Hyper-Connections (mHC)
  • Muon Optimizer
  • FP4 Quantization-Aware Training
  • Pre-Training
  • Post-Training: Specialist Training and On-Policy Distillation
  • Reasoning Modes
  • Putting It All Together
  • Quick Summary

In this lesson

  1. The big picture
  2. Two models: DeepSeek-V4-Pro and DeepSeek-V4-Flash
  3. Hybrid attention with CSA and HCA
  4. Manifold-Constrained Hyper-Connections (mHC)
  5. Muon optimizer
  6. FP4 quantization-aware training
  7. Pre-training
  8. Post-training: specialist training and on-policy distillation
  9. Reasoning modes
  10. Putting it all together
  11. Worked example, step by step
  12. Practice: try it yourself
  13. Quick summary

Key terms in this lesson

CSA
Compressed Sparse Attention: pool every 4 tokens into one KV entry and attend to the top-k selected entries.
HCA
Heavily Compressed Attention: pool every 128 tokens into one KV entry and attend to all of them.
Lightning indexer
A cheap scoring module that picks which compressed entries each query attends to.
mHC
Manifold-Constrained Hyper-Connections: multi-stream residuals with doubly stochastic mixing matrices.
Muon
An optimizer that orthogonalises momentum updates for weight matrices via Newton-Schulz iterations.
Quantization-aware training
Training while simulating low-precision weights so the model stays accurate when quantized.
On-policy distillation
A student learns from teacher feedback on answers the student itself generated.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 6.6 Flash Attention: Memory-Efficient Attention at Scale · 7.1 Small Language Models: Big Capability in Compact Form →