computer use

Most tools hand an agent a clean programming interface. Computer use takes the opposite approach: instead of an API, the agent is given a screen, a mouse, and a keyboard, and made to operate ordinary software the way a person does — looking at the pixels, moving the cursor, clicking buttons, and typing into fields. It lets an agent drive apps that were never built to be automated, because almost everything has a screen even when it has no API.

The loop is perceive-then-act on a graphical interface: the model receives a screenshot, decides on a low-level action like 'click at this coordinate' or 'type this text', the action is performed, and a fresh screenshot comes back so it can see what changed. Doing this well demands reading cluttered interfaces, locating the right element, and chaining many tiny steps without losing track of the goal.

It is powerful but currently slow and brittle — a moved button, a popup, or a slow-loading page can derail a run, and a wrong click can do real damage. Because it acts directly on a live machine, computer use is one of the places where human oversight and tight limits matter most.