Modern AI Engineering

Lesson 8.3 · 24 min · video lesson

Prefix Tuning: Learnable Context Prepended to the Input

Can we teach a frozen language model a new task by training nothing but a handful of invisible “words” placed in front of every input?

In short: Prefix tuning keeps every weight of a pretrained model frozen and learns a short sequence of continuous vectors, the prefix, that is prepended to the keys and values at every attention layer. Real tokens attend to this prefix, which steers the model toward the task. Only around 0.1% of the parameters are trained, and one model can serve many tasks by swapping prefixes.

What you will learn

  • What is a large language model?
  • The problem: why full fine-tuning is expensive
  • What is Prefix Tuning?
  • Prefix Tuning = Prefix + Tuning
  • How does Prefix Tuning work?
  • The prefix is not real words
  • Where the prefix is added
  • How the prefix is trained
  • How small is the prefix really?
  • A simple code example
  • Prefix Tuning vs Full Fine-Tuning
  • Prefix Tuning vs Prompt Tuning
  • Advantages of Prefix Tuning
  • Limitations of Prefix Tuning
  • Where Prefix Tuning is used

In this lesson

  1. What is a large language model? (a quick refresher)
  2. The problem: why full fine-tuning is expensive
  3. What is prefix tuning?
  4. How prefix tuning works
  5. The prefix is not real words
  6. Where the prefix is added and how it is trained
  7. How small is the prefix really?
  8. Prefix tuning vs full fine-tuning vs prompt tuning
  9. Advantages, limitations, and where it is used
  10. Worked example, step by step
  11. Practice: try it yourself

Key terms in this lesson

Prefix
Learned key/value vectors placed before the real tokens at each attention layer.
Soft prompt
A prompt made of trainable continuous vectors instead of real tokens.
Prompt tuning
Learning soft vectors only at the input embedding layer of a frozen model.
Key and value
Vectors each position offers in attention: the key is matched against queries, the value is passed on.
Reparameterisation
Producing the prefix through a small MLP during training for stability, then keeping only the result.
Prefix length (L)
The number of virtual positions in the prefix.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 8.2 LoRA: Parameter-Efficient Fine-Tuning via Low-Rank Matrices · 8.4 Knowledge Distillation: Compressing Large Models into Small Ones →