Modern AI Engineering

Lesson 11.16 · 23 min · video lesson

Computer-Use Agents: Controlling Interfaces with AI

How can an AI that only reads text and images log into a website, fill in a form and click "Submit", using the same screen, mouse and keyboard we do?

In short: A computer-use agent is an AI agent that operates a computer through its graphical interface: it looks at screenshots, decides what to do, and sends mouse and keyboard actions, in a loop until the task is done. It lets AI work with software that has no API, but it is slower, more error-prone and riskier than calling an API, so it needs a sandbox, step limits, human confirmation for important actions and defences against instructions hidden on screen.

What you will learn

  • What is a computer-use agent?
  • Why do we need a computer-use agent?
  • The perceive, think, act loop
  • How does the agent see the screen?
  • How does the agent decide what to do?
  • How does the agent take actions?
  • A step-by-step walkthrough with an example
  • The system prompt and tools
  • Safety and guardrails
  • Limitations of computer-use agents
  • Conclusion

In this lesson

  1. What is a computer-use agent?
  2. Why do we need a computer-use agent?
  3. The perceive, think, act loop
  4. How does the agent see the screen?
  5. How does the agent decide what to do?
  6. How does the agent take actions?
  7. A step-by-step walkthrough with an example
  8. The system prompt and tools
  9. Safety and guardrails
  10. Limitations of computer-use agents, and conclusion
  11. Common mistakes and how to spot them
  12. Practice: try it yourself

Key terms in this lesson

Computer-use agent
An AI agent that controls a computer through its graphical interface using screenshots, mouse and keyboard.
Vision-language model (VLM)
A model that takes images and text as input and produces text, including structured actions.
Agent harness
The code around the model that takes screenshots, sends them to the model and executes its actions.
Grounding
Mapping a description like "the Save button" to an exact location on the screen.
Accessibility tree
A structured list of on-screen UI elements that the OS or browser exposes for assistive technology.
Prompt injection
Malicious instructions hidden in content the agent reads, aiming to override its real task.
RPA
Robotic process automation: scripted, fixed sequences of clicks and keystrokes.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 11.15 Sakana Fugu: Lessons from an Open-Source Agent Study · 12.1 Harness Engineering: The Scaffolding Around AI Agents →