macos-computer-use

Use the computer_use tool: first call capture with mode=som to get a screenshot with numbered element overlays, then call click or type referencing the element index. Re-capture after state-changing actions to verify results.

Element indices come from the accessibility tree and remain reliable across models, while pixel coordinates are fragile when UI layouts shift. The skill recommends clicking by element index and re-capturing whenever the UI changes.

Yes, the workflow is model-agnostic and works with any tool-capable model including GPT, Gemini, and local models via OpenAI-compatible endpoints. There is no Anthropic-specific schema required.

Yes, cua-driver routes input to apps without raising windows, moving the cursor, or switching Spaces. You should scope captures to a specific app and avoid raise_window unless the user explicitly requests it.

Use browser_* tools for web automation since they drive a headless Chromium more reliably. Reserve computer_use for tasks requiring the user's actual native Mac apps like Mail, Messages, Finder, or Figma.

The cua-driver dependency is missing. Run hermes tools and enable Computer Use, which installs cua-driver via its upstream script; macOS Accessibility and Screen Recording permissions are also required.