Vercel just handed AI agents your entire computer, for free

Every AI agent that "uses a computer" has been doing it the clumsy way: take a screenshot, guess from the pixels where the button is, click, take another screenshot. Slow, heavy on tokens and prone to misses. Overnight Vercel Labs released a free tool that lets agents see the screen in a completely different way.

It is called agent-desktop and it runs on macOS, Windows and Linux. Instead of a picture, the agent reads the operating system's accessibility tree, the same structure screen readers use. The tool hands the agent clean JSON with every button, field and menu item, each with its own reference. The agent acts on a reference, takes a fresh snapshot and checks that it worked. No pixel guessing at all.

There are a couple of nice tricks too. In dense apps like Slack the agent first looks at a rough map of the window and then drills into the one region it needs, which saves a lot of tokens. By default everything runs in the background: the pointer doesn't move and focus isn't stolen, so you can keep working on the same machine. In multi-agent mode each agent gets its own labelled cursor, so you can see who is doing what.

Install it with npm i -g agent-desktop; there is a ready-made skill for Claude Code, Codex and Cursor. It ships around sixty commands covering clicks, typing, windows, clipboard, notifications and screenshots. The same team already has agent-browser for the web, so together they cover websites and regular apps. The code is open under Apache 2.0 and the repo already has more than three thousand stars on GitHub. On macOS you need to grant Accessibility permission, and the authors openly call it a Labs experiment.

Verdict: agents finally got hands instead of a magnifying glass. Try it on something you don't mind breaking first.