Hermes Computer Use: Setup, CUA Driver, Doctor | Easy Understand Hermes Agent

Imagine you’re deep in a report, and your AI assistant is quietly reorganizing folders, filling out forms, and scrolling web pages on a “virtual screen” beside you—like an invisible coworker who never interrupts your flow.

Quick answer. Enable Computer Use with hermes computer-use install, check the install with hermes computer-use status, and diagnose anything that isn’t working with hermes computer-use doctor. Then start a session that has the toolset loaded: hermes -t computer_use chat.

The commands, back to back

What you want	Command
Install the CUA driver (older installs, or a manual repair)	`hermes computer-use install`
Verify the install	`hermes computer-use status`
Find out why a click or keystroke isn’t landing	`hermes computer-use doctor`
Start a session with Computer Use enabled	`hermes -t computer_use chat`
Enable it in every session	add `computer_use` to `toolsets` in `~/.hermes/config.yaml`

What hermes computer-use doctor prints

It runs cua-driver’s structured health_report check and prints one line per check. Your version string appears where this example shows <version>:

$ hermes computer-use doctor
⚠️  cua-driver <version> on darwin: degraded

  ✅ binary_version: cua-driver <version>
  ✅ platform_supported: macOS 26.4.1 (arm64)
  ✅ session_active: MCP session is active.
  ❌ bundle_identity: Process has no CFBundleIdentifier.
      → Run the binary inside CuaDriver.app so TCC grants attribute correctly.
  ✅ tcc_accessibility: Accessibility is granted.
  ✅ tcc_screen_recording: Screen Recording is granted.
  ✅ ax_capability: AX is trusted and reachable.
  ✅ screen_capture_capability: ScreenCaptureKit reachable; 1 display(s) shareable.

Reading it: exit code 0 means the overall status is ok; 1 means degraded or failed (at least one ❌, and the indented hint on that line tells you what to fix); 2 means the cua-driver binary itself isn’t reachable. Add --include CHECK / --skip CHECK to narrow the run, or --json for the raw payload. On Windows and Linux, bundle_identity and the tcc_* checks report skip — those are macOS-only concepts, not failures. On Linux, where the daemon is a hand-written systemd user unit or XDG autostart entry rather than a managed autostart, doctor also reads those units: a cua-driver ExecStart pointing at a pruned packages/releases/<version>/ directory is reported as a failing daemon unit (...) check (point it at ~/.cua-driver/packages/current/cua-driver), and a unit that runs cua-driver serve gets a daemon (...) check that connects to its socket — fail when nothing is listening (crash loop, stopped, never started), pass when the daemon answers. Reinstalling the driver does not start a daemon; systemctl --user status <unit> does. hermes computer-use status prints the same dead-daemon line and exits 1.

That’s exactly what Computer Use in Hermes Agent does. It lets the AI see your screen content and simulate clicks, typing, scrolling, and dragging—without moving your mouse, stealing keyboard focus, or switching your virtual desktop. You and the AI work side‑by‑side on the same machine, completely independent.

Which AI Models Work?

Almost any model that can call tools—Claude, GPT, Gemini, even local open‑source models. Hermes doesn’t lock you into a proprietary format. No special Anthropic syntax to learn. Just plug in your favorite model and go.

Which Machine Does It Drive?

computer_use acts on the machine where the bot’s screen lives — never on the machine running Hermes Desktop. On a gateway with terminal.backend: local, that’s the gateway host. With a sandboxed terminal (docker, ssh, singularity), the driver runs inside the sandbox on the sandbox’s own display, so it can only touch what the terminal can — and the sandbox image must carry cua-driver (nousresearch/hermes-sandbox:desktop does). Modal, Daytona and Vercel sandboxes can’t host a display yet, so with those backends computer_use refuses unless bot_desktop.placement: gateway opts into driving the host. Details: Bot Screen → Where the screen runs.

How Does It Pull This Off?

Hermes uses an open‑source driver called cua-driver. It reads on‑screen elements (buttons, input boxes, menus) through accessibility interfaces, then sends simulated actions to the target window. Under the hood, each OS uses its native accessibility layer:

macOS: AX (Accessibility) — sends events without moving the real cursor
Windows: UIAutomation — inputs don’t steal focus
Linux: AT‑SPI (X11 & Wayland) — stays out of your foreground

The core principle: never force a window to the foreground. Curious why that matters? See the no‑foreground contract.

Enable It — Two Ways

The driver ships with Hermes. cua-driver is pinned in pm/lock.json and is a default PM package: the installers, a bare hermes pm install, and hermes update install it on every macOS, Windows, and glibc Linux target (cua-driver publishes no musl or Android build). The desktop app’s bundle carries it too. To leave it out, pass --skip-computer-use on POSIX or -SkipComputerUse on Windows (or run hermes pm install --without cua-driver); Hermes remembers the choice, and hermes pm install cua-driver undoes it.

Option 1: Direct command line (easiest)

hermes computer-use install

This asks PM to prepare the pinned cua-driver package (verified against pm/lock.json) — it does not run the upstream installer. Use hermes computer-use status to verify the install:

hermes computer-use status

Option 2: Interactive setup

hermes tools

Select 🖱️ Computer Use (macOS/Windows/Linux) from the list—it runs the same installer automatically. The dashboard / desktop app toggle works the same way: if the driver is missing, the toggle kicks off the install in the background (watch progress in the toolset panel).

If you already have cua-driver, Hermes reuses it when it supports the 0.20 runtime contract. During setup, toolset enablement, hermes update, and the first computer_use call of a session, Hermes checks the local version and manifest, and repairs an old or incomplete standard installation through PM (at most once per session at runtime). A binary you selected with HERMES_CUA_DRIVER_CMD stays under your control, so Hermes reports the incompatibility and leaves it unchanged.

Swapping the driver. computer_use.backend in config.yaml names the one active provider — cua (the default) or an installed provider plugin. See computer-use backend plugins. hermes tools → Computer Use lists every installed provider.

Post‑Install Permissions

Give your terminal (or Hermes App) a couple of permissions:

macOS: System Settings → Privacy & Security → enable Accessibility and Screen Recording. Grant the identity named by hermes computer-use doctor (CuaDriver, com.trycua.driver, in every permission mode — the driver daemon always launches through CuaDriver.app).
Windows: Local use needs nothing extra. For SSH remote access, follow the Windows‑SSH guide.
Linux: Set DISPLAY (X11) or XDG_SESSION_TYPE=wayland (Wayland needs XWayland bridge). Ensure AT‑SPI is running (GNOME/KDE enable it by default).

Start Using It

After install & permissions, launch a chat with the toolset:

hermes -t computer_use chat

Or add computer_use to ~/.hermes/config.yaml under toolsets to auto‑enable it every session.

Hermes maps its existing approval system onto cua‑driver’s immutable daemon mode—so there’s no second permission toggle to remember. The same approve/deny flow you already know applies. Before the AI touches your screen, it asks you first. Safe and simple.

Permission mode and capability manifest approval are launch settings—they can’t change once the runtime starts. For repeatable automation (cron jobs, scheduled research against a signed-in app), bounded mode lets you review a capability manifest once, and anything outside it fails closed. Bounded and unrestricted modes use a private embedded daemon, launched through CuaDriver.app on macOS (see above).

Destructive actions (click, type, drag, scroll, key, focus_app) go through the same approval gate as dangerous shell commands, and where nobody can answer—cron, single-query, unattended platforms, or any headless run—the action is refused rather than auto-approved (--yolo / /yolo still bypasses).

Browser work—including pages in a signed-in profile—goes through the browser toolset (browser_exec), not computer_use.

Summary & Practical Tips

Bottom line: Computer Use = the AI works on a “background copy” of your screen while you keep doing your thing.

Pro tip: Before your first real task, run hermes computer-use doctor to verify permissions. If clicks aren’t landing, 90% of the time it’s missing Screen Recording or Accessibility access—don’t reinstall, just check the diagnostics. Also, test typing and scrolling in a blank notepad window first. Get comfortable there, then let it loose on real apps.