中文版

agent-desktop（GitHub：https://github.com/vercel-labs/agent-desktop ，2026-02-19 创建，今晨抓取时已 9,709★）是 Vercel Labs 出品的 Computer use CLI for AI agents：让 AI agent 通过操作系统的无障碍树（accessibility tree）读写桌面应用。

它的工作方式是“快照—操作—再快照”：一次快照返回 JSON，每个可操作元素带一个 ref；agent 对 ref 执行点击、输入等操作，再快照一次核对结果，全程不做像素匹配。CLI 与 C 库覆盖 macOS、Windows、Linux，命令分观察、按 ref 操作、键盘鼠标、应用与窗口、通知、剪贴板、等待、会话、系统等组，共 60 个命令名；自带 skill（agent-desktop skills get desktop / platform）告诉 agent 循环规则与各平台差异。安装用 npm install -g agent-desktop，macOS 需要开 Accessibility 权限，Apache-2.0 开源。

为什么值得了解

• 做电脑操作 agent 的人最头疼的就是“截图坐标一变就点歪”；走无障碍树 + ref 的路线比像素匹配稳得多，这个项目把这套做法做成了一个开箱即用的 CLI
• “操作后必须再快照核对”的循环值得所有 agent 开发者学：agent 不是发完命令就完事，验证结果才是闭环
• README 也提醒了隐私边界：快照、剪贴板读取、截图与命令输出可能包含私密桌面内容，不要放进公开日志和上传内容里

English Version

agent-desktop (GitHub: https://github.com/vercel-labs/agent-desktop , created 2026-02-19, 9,709 stars at this morning’s fetch) is Vercel Labs’ Computer use CLI for AI agents: it lets an AI agent read and operate desktop apps through the operating system’s accessibility tree.

It works as “snapshot — act — snapshot again”: one snapshot returns JSON with a ref for every element an agent can act on; the agent acts on a ref (click, type and so on) and snapshots again to verify the result, with no pixel matching anywhere in the loop. The CLI and C library cover macOS, Windows and Linux, with 60 command names across observe, act-on-a-ref, keyboard and mouse, apps and windows, notifications, clipboard, wait, session and system groups; built-in skills (agent-desktop skills get desktop / platform) teach an agent the loop rules and per-platform differences. Install with npm install -g agent-desktop; macOS needs Accessibility permission. Apache-2.0 licensed.

Why it’s worth knowing

• Anyone building computer-use agents knows the pain of “the screenshot coordinates shifted and the click missed”; going through the accessibility tree with refs is far more robust than pixel matching, and this project packages that approach as a ready-to-use CLI
• The “snapshot again after acting to verify” loop is worth learning for every agent developer: issuing a command is not the end of the job, verifying the result closes the loop
• The README also draws a privacy line worth respecting: snapshots, clipboard reads, screenshots and command output can contain private desktop content, so keep them out of public logs and uploads