Lesson 4.14 · 27 min · video lesson
Rotary Position Encoding: Position Without Fixed Lookup Tables
Attention on its own cannot tell "dog bites man" from "man bites dog". How do modern LLMs know word order, using nothing but a rotation?
In short: RoPE (Rotary Position Embedding) encodes position by rotating each query and key vector by an angle proportional to the token's position, using different rotation speeds for different pairs of dimensions. Because rotating both vectors and then taking a dot product only depends on the difference of their angles, the attention score automatically depends on the relative distance between tokens. It adds no parameters, keeps vector lengths unchanged, and is used by most open LLMs today.
What you will learn
- The Big Picture
- Why a Transformer Needs Position Information
- Older Approaches and Their Problems
- The Core Idea Behind RoPE
- The 2D Rotation Math
- How RoPE Is Applied to Q and K
- Why the Dot Product Captures Relative Position
- A Small Numeric Example
- Real-World Use Cases
- Quick Summary
In this lesson
- The big picture
- Why a Transformer needs position information
- Older approaches and their problems
- The core idea behind RoPE
- The 2D rotation math
- How RoPE is applied to Q and K
- Why the dot product captures relative position
- A small numeric example
- Comparison, real-world use and pitfalls
- Going one level deeper
- Practice: try it yourself
Key terms in this lesson
- RoPE
- Rotary Position Embedding: encoding position by rotating query and key vectors.
- Rotation matrix
- A matrix that turns a vector by an angle without changing its length.
- Frequency θᵢ
- How many radians pair i rotates per position; base^(−2i/d).
- Relative position
- The distance between two tokens, as opposed to their index in the sequence.
- Permutation-equivariant
- Shuffling the inputs simply shuffles the outputs the same way, so order is invisible.
- ALiBi
- A position method that subtracts a distance-proportional penalty from attention scores.
- Position interpolation
- Extending context by squeezing new positions into the angle range seen during training.
This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).
← 4.13 Cross-Attention: Connecting Encoder Output to the Decoder · 4.15 Feed-Forward Networks: The Transformer's Memory Layer →