macos-computer-use

Automates macOS desktop interactions via screenshots, element indexing, mouse, and keyboard control.

Updated Jun 17, 2026
One-click install
npx skills add https://github.com/i-bebsi/hermes-agent --skill macos-computer-use-i-bebsi
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: macos-computer-use
Source: https://github.com/i-bebsi/hermes-agent/tree/main/hermes-config/skills/apple/macos-computer-use
Command: npx skills add https://github.com/i-bebsi/hermes-agent --skill macos-computer-use-i-bebsi

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Automating native macOS applications normally requires stealing the user's cursor and keyboard focus, interrupting their work. This Skill drives the Mac desktop in the background — clicking, typing, scrolling, and dragging in any app or Space without disrupting the user, and works with any tool-capable model. ## Core Features & Use Cases - Background GUI Automation: Capture screenshots with numbered element overlays (SOM mode) and click by element index instead of unreliable pixel coordinates. - Full Input Control: Supports click, double-click, right-click, drag-and-drop, scroll, text typing, and keyboard shortcuts like cmd+s, all with optional post-action capture for verification. - Safety Guardrails: Hard-blocks dangerous patterns (log out, lock screen, fork bombs), refuses to type secrets, and treats on-screen content as untrusted to prevent prompt injection. - Use Case: Ask the agent to organize files in Finder or reply in the native Mail app while you keep typing in your editor — the automation runs in another Space without touching your focus. ## Quick Start Use the computer_use tool to capture the Safari window and click the Sign In button by its element index.

Frequently Asked Questions about macos-computer-use

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate macOS apps without moving the user's cursor?▼

Use the computer_use tool, which routes input to apps in the background without raising windows or switching Spaces. Capture with mode="som" to get numbered element overlays, then click by element index rather than coordinates.

How to click UI elements reliably with an AI agent on macOS?▼

Capture the screen first with computer_use(action="capture", mode="som") to get an AX-tree index of interactable elements, then call click with the element number. Element indices are far more reliable than pixel coordinates across models.

Does macOS computer use automation work with models other than Claude?▼

Yes, it works with any tool-capable model including GPT, Gemini, or local models via OpenAI-compatible endpoints. There is no Anthropic-native schema; all actions use a universal tool interface.

When should I use computer_use instead of browser automation tools?▼

Use browser_* tools for web tasks since headless Chromium is more reliable than driving a GUI browser. Reserve computer_use for native Mac apps like Mail, Messages, Finder, Figma, or anything non-web.

Why does computer_use report cua-driver not installed?▼

The cua-driver dependency is missing. Run hermes tools and enable Computer Use, which installs cua-driver via its upstream script. macOS Accessibility and Screen Recording permissions are also required.

What actions are blocked by macOS computer use safety rules?▼

The tool hard-blocks log out, lock screen, force empty trash, and dangerous typed commands like curl piped to bash. It also refuses to type passwords, API keys, or credit card numbers, and will not click permission or payment dialogs.