computer-use agents

A computer-use agent operates software the way a person does: it looks at the screen, decides where to click and what to type, and issues mouse and keyboard actions. Instead of relying on every app exposing a clean API, it drives the graphical interface directly, so it can in principle use any program a human can. Think of an assistant that, given a goal, takes over the cursor and works the application on your behalf.

The control loop is perception then action: the agent receives a screenshot (sometimes augmented with the accessibility tree or set-of-marks labels that tag clickable elements), a multimodal model grounds the goal to a specific UI element and predicts an action — click at these coordinates, type this text, scroll, press a key — and the environment renders the next screen, which feeds the following step. This is embodied interaction in a digital world, requiring visual grounding, spatial reasoning, and recovery from unexpected dialogs or layout changes.

Pixel-level GUI control is general but brittle and slow: coordinates miss, pages shift, latency compounds over many steps, and benchmarks show success rates still well below human reliability on long workflows. It also concentrates risk — an agent with screen and keyboard control can take consequential, hard-to-undo actions — so confirmations, restricted environments, and human oversight remain essential.