macos-computer-use

Use the computer_use tool, which routes input to apps in the background without raising windows or switching Spaces. Capture the target app first, then click elements by index so the user can keep working uninterrupted.

Capture the screen in SOM mode to get numbered overlays on every interactable element, then call click with the element index. Index-based clicking is far more reliable than pixel coordinates, especially for non-Claude models.

Yes, it works with any tool-capable model including GPT, Gemini, and open models on local OpenAI-compatible endpoints. There is no Anthropic-native schema; all actions use plain parameters like action, element, and coordinate.

The cua-driver dependency is missing. Run hermes tools and enable Computer Use, which installs cua-driver via its upstream script. macOS Accessibility and Screen Recording permissions are also required.

Use browser_* tools for web automation since they run a real headless Chromium and are more reliable. Reserve computer_use for tasks requiring the user's actual native Mac apps like Mail, Messages, Finder, or Figma.

The element index may be stale because the UI changed after your last capture, or a modal dialog is blocking input. Re-capture the screen, dismiss any modal with escape or its close button, then retry the click.