Modern AI Engineering

Lesson 4.15 · 27 min · video lesson

Feed-Forward Networks: The Transformer's Memory Layer

Attention gets all the fame, yet about two thirds of a typical LLM's parameters sit somewhere else. What is that "somewhere else" doing?

In short: Every Transformer layer has two parts: attention, which moves information between tokens, and a feed-forward network (FFN), which processes each token on its own. The FFN expands each token vector to a wider hidden layer (typically about 4× wider, or about 2.7× with three matrices in SwiGLU), applies a non-linear activation, and contracts it back. It holds most of the parameters and is widely believed to store much of the model's learned knowledge; in Mixture-of-Experts models, it is the part that gets split into experts.

What you will learn

  • What is a Feed-Forward Network?
  • Understanding Feed-Forward Networks with a Real-World Analogy
  • Where Does the Feed-Forward Network Sit in a Transformer?
  • How Does a Feed-Forward Network Work - Step by Step
  • The Expand-then-Contract Pattern
  • Why Does the FFN Expand and Then Contract?
  • ReLU and Activation Functions
  • What Does the Feed-Forward Network Actually Learn?
  • How Much of the Model is the Feed-Forward Network?
  • Feed-Forward Networks in Mixture of Experts
  • Why Feed-Forward Networks Are So Important
  • [Softmax Activation Function in Machine Learning](https://www.youtube.com/watch?v=2Zx6x01WwWM) (Video)
  • Inside ChatGPT: What happens token by token
  • [Tokenization in Large Language Models (LLMs)](https://www.youtube.com/watch?v=sK2s9I84EVI) (Video)
  • [Embeddings in Machine Learning](https://www.youtube.com/watch?v=LedXW6xl21s) (Video)
  • Positional embeddings: how transformers track order

In this lesson

  1. What is a feed-forward network?
  2. Understanding the FFN with a real-world analogy
  3. Where does the FFN sit in a Transformer?
  4. How a feed-forward network works, step by step
  5. The expand-then-contract pattern, and why
  6. ReLU and other activation functions
  7. Code you can run
  8. What does the FFN actually learn?
  9. How much of the model is the FFN, and FFNs in Mixture of Experts
  10. Why FFNs are so important, common mistakes and further resources
  11. Worked example, step by step
  12. Practice: try it yourself

Key terms in this lesson

Feed-forward network (FFN)
A network where data flows one way through layers; in Transformers, a two-layer MLP applied to each token.
d_ff
The width of the FFN's hidden layer, usually several times d_model.
Activation function
A non-linear function such as ReLU, GELU or SiLU applied to each hidden value.
SwiGLU
A gated FFN variant: a SiLU-activated gate multiplies an up projection before the down projection.
Residual connection
Adding a sub-layer's input to its output, so each sub-layer writes an update.
Mixture of Experts (MoE)
Replacing one FFN with many expert FFNs and a router that picks a few per token.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 4.14 Rotary Position Encoding: Position Without Fixed Lookup Tables · 5.1 Temperature Sampling: Dialing Up or Down Creativity →