Modern AI Engineering

Lesson 17.5 · 23 min · video lesson

Cloud vs Edge: Where Should Your Model Run?

Should your model live in a data center a thousand kilometres away, or inside the phone in your user's pocket?

In short: Deployment is putting a trained model where it can answer real requests. In cloud deployment the model runs on servers and devices send data over the network; in on-device deployment the model ships inside the app and runs locally. The choice trades model size and easy updates (cloud) against latency, privacy, offline use and zero server cost (on-device), and many real products use a hybrid of both.

What you will learn

  • What is deployment?
  • Training and inference
  • What is Cloud Deployment?
  • What is On-device Deployment?
  • The one big difference
  • The round trip problem
  • Where does our data go?
  • How big can the model be?
  • Who pays the bill?
  • The shipping problem
  • What happens when the network is gone?
  • The hybrid approach
  • Some real examples
  • Let's tabulate the difference
  • When to use which one?
  • Summary

In this lesson

  1. What is deployment? Training vs inference
  2. Cloud deployment and on-device deployment
  3. The round trip problem and where our data goes
  4. Model size, the bill, and the shipping problem
  5. Code: putting numbers on the trade-off
  6. The hybrid approach
  7. Some real examples
  8. Let's tabulate the difference, and when to use which
  9. Worked example, step by step
  10. Practice: try it yourself
  11. Common mistakes and limits

Key terms in this lesson

Inference
Using a trained model to produce an output for a new input.
Deployment
Putting a trained model somewhere it can serve inference requests from real users.
On-device (edge) deployment
Running the model locally on the user's phone, laptop or embedded hardware.
Round-trip time (RTT)
The time for a small message to travel to a server and for the reply to come back.
Quantisation
Storing model weights with fewer bits (e.g. 8-bit integers) to shrink memory and speed up inference.
NPU
Neural processing unit: a chip on many phones built to run neural network math efficiently.
Hybrid deployment
Combining a small on-device model with a larger cloud model, escalating only when needed.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 17.4 Language Processing Units: A New Approach to LLM Inference · 17.6 On-Device ML: A TensorFlow Lite Android Walkthrough →