WTF is Computer Use?
Agents that operate a computer like a person: looking at the screen, moving the mouse and typing.
Also called: computer-use agent, cua, browser use
TL;DR
Instead of needing an API for everything, a computer-use agent takes screenshots, decides where to click, and types into apps and websites. It is slower and less reliable than APIs, but works with software that was never built for AI.
How it works
The model receives screenshots (sometimes plus accessibility trees or DOM data) and emits actions such as click(x, y), type(text), scroll or key presses through a special tool. A harness executes them in a sandboxed VM or browser and returns the new screenshot. Main challenges are latency, visual grounding (clicking the right pixel), and safety, since on-screen text can carry prompt injections.
Origin
- Coined / introduced by
- Anthropic (first major release)
- When
- 22 Oct 2024
- Story
- Launched in public beta by Anthropic; other vendors shipped similar capabilities soon after.