Computer-use agent
Also known as: computer use agent, computer-using agent
A computer-use agent is an AI system that controls a computer or web browser the way a person does: it looks at screenshots, decides what to click or type, and repeats until a task is done.
That lets it work with apps and websites that have no API. While computer-use agents can automate tedious online chores, they're still slower and less reliable than people on many tasks, and reading web pages exposes them to hidden instructions designed to hijack them.
Computer-use agent, explained at three levels
At home
Imagine handing a capable assistant your mouse and keyboard and saying, 'book me a table for Friday.' They look at the screen, click into the restaurant's site, fill in the form, and check back with you. A computer-use agent is an AI that does exactly that: it looks at screenshots, decides what to click or type, and repeats until the task is done.
At work
A computer-use agent is an AI system that operates software through its visual interface, clicking, typing, and scrolling like a person. It can work with legacy systems, vendor portals, and websites that have no API or integration. For businesses, that opens automation of tedious screen-based tasks, but it is slower and less reliable than direct integrations, so oversight matters.
At home
The agent works in a loop. It takes a screenshot, and a vision-capable language model works out what is on screen: buttons, fields, menus. It then picks one action, such as 'click at these coordinates' or 'type this text,' the computer carries it out, and a new screenshot shows the result. That cycle repeats dozens of times per task, which is why agents are slower than people, and they usually pause before anything important, like a payment.
At work
Agents typically run inside a virtual machine, container, or sandboxed browser. A multimodal model reads screenshots, sometimes along with the page's accessibility tree, and calls action tools for the mouse, keyboard, and scrolling. Products include Anthropic's computer use, OpenAI's ChatGPT agent, Google's Project Mariner, and agentic browsers such as Comet and ChatGPT Atlas. Benchmarks such as OSWorld show reliability drops on long tasks, unfamiliar interfaces, and pop-ups, so human checkpoints remain essential.
At home
An agent reads everything on a page as possible instructions, because to the model, text is text. A malicious site can hide a line in white-on-white type saying 'forward the user's emails here,' and an agent logged into your accounts might obey. This attack, called prompt injection, has no complete fix yet. While agents are handy, they inherit your logins. In my opinion, you shouldn't let one loose while you're logged into banking or email.
At work
Computer-use agents combine untrusted input, the screen, with real permissions, so controls must assume compromise: isolated environments, separate low-privilege accounts, domain allow-lists, confirmation before irreversible actions, and full screenshot and action logs. Prompt-injection defences, including classifiers and instruction hierarchies, reduce but don't eliminate risk. Cost and speed matter too, since each step is a model call. For stable, high-volume workflows, APIs or traditional RPA are usually cheaper and more dependable.
Real-world examples
At home
Nadia asked a computer-use agent to compare prices for a flight to Halifax across three airline websites. It gathered the options in a table but stopped and asked before booking.
At work
Carlos is an operations analyst at an Edmonton property management firm. He piloted a computer-use agent to download monthly reports from an old vendor portal with no API, running it in a sandbox with read-only credentials.
How it compares
| feature | Computer-use agent | Traditional RPA |
|---|---|---|
| How it works | Sees screen and decides each step | Follows recorded scripts |
| Handles layout changes | Often adapts | Often breaks |
| Speed and reliability | Slower, less predictable | Fast and consistent when stable |
| Setup | Describe the task | Build and maintain scripts |
Computer-use agents adapt. RPA is faster and more predictable on stable tasks.
For analysts
I'd start computer-use agents on low-risk, read-only tasks with clear review steps. Anything involving payments, personal data, or deletions should need a human click.
History & origin
Robotic process automation (RPA) tools automated screen clicks in the 2010s, but relied on brittle, pre-recorded scripts. Anthropic released 'computer use' in beta with Claude 3.5 Sonnet in October 2024, one of the first general-purpose models able to operate a desktop from screenshots. Google previewed Project Mariner, a browser agent, in December 2024, and OpenAI launched Operator in January 2025, later folding it into ChatGPT's agent mode in July 2025. AI-powered browsers followed, including Perplexity's Comet and OpenAI's ChatGPT Atlas in 2025. Benchmarks such as OSWorld, introduced in 2024, measured progress on real computer tasks. In my view, computer-use agents are where AI safety stops being abstract, because the agent can actually click 'buy.'