Modern AI Engineering

Lesson 8.2 · 25 min · video lesson

LoRA: Parameter-Efficient Fine-Tuning via Low-Rank Matrices

What if we could fine-tune a 7-billion-parameter model by training only about 4 million numbers, and get nearly the same result?

In short: LoRA (Low-Rank Adaptation) freezes a pretrained model’s weights and learns the change to each chosen weight matrix as the product of two thin matrices, B·A. Because the update is low-rank, it needs a tiny fraction of the trainable parameters and optimizer memory. After training, B·A can be merged back into W, so inference costs nothing extra.

What you will learn

  • The Big Picture
  • Why Full Fine-Tuning Is Expensive
  • The Core Idea Behind LoRA
  • How LoRA Works Step by Step
  • A Small Numeric Example
  • Where LoRA Is Applied in a Transformer
  • Merging LoRA Back Into the Model
  • Real-World Use Cases
  • Quick Summary

In this lesson

  1. The big picture
  2. Why full fine-tuning is expensive
  3. The core idea behind LoRA
  4. How LoRA works step by step
  5. A small numeric example
  6. Where LoRA is applied in a Transformer
  7. Merging LoRA back into the model
  8. Real-world use cases and limits
  9. Common mistakes and how to spot them
  10. Practice: try it yourself
  11. Quick summary

Key terms in this lesson

LoRA
Low-Rank Adaptation: training a low-rank update B·A next to frozen pretrained weights.
Rank (r)
The number of independent directions in a matrix; in LoRA, the inner size of B and A.
Frozen weights
Parameters that are not updated during training.
Adapter
A small set of trainable weights added to a frozen model for one task.
Alpha (α)
A LoRA scaling constant; the adapter output is multiplied by α/r.
Merging
Adding the trained B·A into W so the model has its original shape and speed.
QLoRA
LoRA training on top of a 4-bit quantized frozen base model.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 8.1 Fine-Tuning: Adapting a Pre-Trained Model to Your Task · 8.3 Prefix Tuning: Learnable Context Prepended to the Input →