Modern AI Engineering

Lesson 5.3 · 23 min · video lesson

Token Streaming: Rendering Outputs as They Arrive

A long chatbot answer can take 20 seconds to generate, yet the first words appear almost instantly. How does text travel to your screen while the model is still writing it?

In short: LLMs generate one token at a time, so a server can send each token as soon as it exists instead of waiting for the full answer. Most LLM APIs do this with Server-Sent Events (SSE): one long-lived HTTP response with content type text/event-stream, in which each token arrives as a small "data:" event followed by a blank line, and a final marker (such as data: [DONE]) ends the stream. Streaming does not make generation faster; it makes the wait feel much shorter by cutting time to first token.

What you will learn

  • What is token streaming
  • A quick recap of how an LLM generates text
  • Why we need streaming at all
  • What is SSE
  • How the HTTP connection stays open
  • The format of a streamed message
  • A full walkthrough from server to screen
  • The [DONE] marker that ends the stream
  • SSE vs WebSockets
  • Token streaming in the real world

In this lesson

  1. What is token streaming?
  2. A quick recap of how an LLM generates text
  3. Why we need streaming at all
  4. What is SSE, and how the connection stays open
  5. The format of a streamed message
  6. A full walkthrough from server to screen
  7. The [DONE] marker that ends the stream
  8. SSE vs WebSockets
  9. Token streaming in the real world, and common mistakes
  10. Worked example, step by step
  11. Practice: try it yourself

Key terms in this lesson

Token streaming
Sending generated tokens to the client as they are produced instead of all at the end.
Time to first token (TTFT)
The delay between sending a request and seeing the first output token.
Server-Sent Events (SSE)
A web standard for pushing a stream of text events from server to client over one HTTP response.
Chunked transfer encoding
An HTTP/1.1 way to send a body of unknown length as a series of sized pieces.
Delta
The new piece of text in a streamed event, to be appended to what came before.
WebSocket
A persistent, two-way connection that lets client and server send messages at any time.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 5.2 Nucleus Sampling: Top-k and Top-p Demystified · 5.4 Lost in the Middle: Why LLMs Miss Central Context →