Computer Use — AI Operates Your PC

Imagine you’re deep in a report, and your AI assistant is quietly reorganizing folders, filling out forms, and scrolling web pages on a “virtual screen” beside you—like an invisible coworker who never interrupts your flow.

That’s exactly what Computer Use in Hermes Agent does. It lets the AI see your screen content and simulate clicks, typing, scrolling, and dragging—without moving your mouse, stealing keyboard focus, or switching your virtual desktop. You and the AI work side‑by‑side on the same machine, completely independent.

Which AI Models Work?

Almost any model that can call tools—Claude, GPT, Gemini, even local open‑source models. Hermes doesn’t lock you into a proprietary format. No special Anthropic syntax to learn. Just plug in your favorite model and go.

How Does It Pull This Off?

Hermes uses an open‑source driver called cua-driver. It reads on‑screen elements (buttons, input boxes, menus) through accessibility interfaces, then sends simulated actions to the target window. Under the hood, each OS uses its native accessibility layer:

• macOS: AX (Accessibility) — sends events without moving the real cursor
• Windows: UIAutomation — inputs don’t steal focus
• Linux: AT‑SPI (X11 & Wayland) — stays out of your foreground

The core principle: never force a window to the foreground. Curious why that matters? See the no‑foreground contract.

Enable It — Two Ways

Option 1: Direct command line (easiest)

hermes computer-use install

Then verify:

hermes computer-use status

Option 2: Interactive setup

hermes tools

Select 🖱️ Computer Use (macOS/Windows/Linux) from the list—it runs the same installer automatically.

Post‑Install Permissions

Give your terminal (or Hermes App) a couple of permissions:

• macOS: System Settings → Privacy & Security → enable Accessibility and Screen Recording. Not sure which? Run hermes computer-use doctor—it tells you exactly what’s missing.
• Windows: Local use needs nothing extra. For SSH remote access, follow the Windows‑SSH guide.
• Linux: Set DISPLAY (X11) or XDG_SESSION_TYPE=wayland (Wayland needs XWayland bridge). Ensure AT‑SPI is running (GNOME/KDE enable it by default).

Start Using It

After install & permissions, launch a chat with the toolset:

hermes -t computer_use chat

Or add computer_use to ~/.hermes/config.yaml under toolsets to auto‑enable it every session.

Permission Mode & Browser Login

Hermes maps its existing approval system onto cua‑driver’s immutable daemon mode—so there’s no second permission toggle to remember. The same approve/deny flow you already know applies. Before the AI touches your screen, it asks you first. Safe and simple.

Summary & Practical Tips

Bottom line: Computer Use = the AI works on a “background copy” of your screen while you keep doing your thing.

Pro tip: Before your first real task, run hermes computer-use doctor to verify permissions. If clicks aren’t landing, 90% of the time it’s missing Screen Recording or Accessibility access—don’t reinstall, just check the diagnostics. Also, test typing and scrolling in a blank notepad window first. Get comfortable there, then let it loose on real apps.