macos-computer-use

Use the computer_use tool, which routes input to apps in the background via cua-driver. Actions like click, type, and scroll target an app's window directly without raising it, stealing keyboard focus, or switching Spaces.

Capture the screen with mode="som" to get numbered overlays on every interactable element plus an accessibility tree index. Then click by element index (element=N) instead of pixel coordinates, which is far more reliable across models.

Yes, the computer_use tool works with any tool-capable model including GPT, Gemini, and local models on OpenAI-compatible endpoints. There is no Anthropic-native schema; element-index clicking is recommended for non-Claude models.

The element index is likely stale because the UI changed after the last capture, or a modal dialog is blocking input. Re-capture the screen to refresh indices, dismiss any blocking dialog with escape or its close button, then retry the click.

Use browser_* tools for web automation since they run a real headless Chromium and are more reliable than driving the user's GUI browser. Reserve computer_use for native Mac apps like Mail, Messages, Finder, Figma, or games.