skills.rest
Reimonsk8
Reimonsk8COMMUNITY
Explore all skills by this author →
Website𝕏@reimonsk8GitHub

macos-computer-use

Automates macOS desktop interactions via screenshots, element indexing, and background input routing.

Updated May 29, 2026
Search Tags:#accessibility#macos#computer use#screenshot#desktop#gui automation#background input
One-click install
npx skills add https://github.com/Reimonsk8/hermes-9router-model-balancing --skill macos-computer-use-reimonsk8
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: macos-computer-use
Source: https://github.com/Reimonsk8/hermes-9router-model-balancing/tree/main/scripts/skills/apple/macos-computer-use
Command: npx skills add https://github.com/Reimonsk8/hermes-9router-model-balancing --skill macos-computer-use-reimonsk8

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Controlling a Mac's GUI normally requires stealing the user's cursor, keyboard focus, or active Space, which interrupts their work. This Skill drives the macOS desktop entirely in the background using the computer_use tool, so automation runs while the user keeps working. ## Core Features & Use Cases - Background GUI Automation: Capture screenshots with numbered element overlays (SOM mode), then click, type, scroll, and drag by element index without raising windows or switching Spaces. - Model-Agnostic Operation: Works with any tool-capable model (Claude, GPT, Gemini, or local OpenAI-compatible endpoints) using a simple action-based API instead of vendor-specific schemas. - Built-in Safety Guardrails: Hard rules block interaction with permission dialogs, password fields, payment UI, and prompt-injection content in screenshots. - Use Case: Ask the agent to organize files in Finder or reply in a native Mac app while you continue typing in your editor in another Space. ## Quick Start Ask the agent to capture the Safari window with computer_use and click a specific element by its index number.

Frequently Asked Questions about macos-computer-use

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate macOS GUI apps without moving the user's cursor?▼

Use the computer_use tool with capture in SOM mode to get numbered element overlays, then click by element index. Input routing works in the background without raising windows, switching Spaces, or stealing keyboard focus.

How to click UI elements reliably with computer vision models on macOS?▼

Capture with mode=som to get an AX-tree index of numbered interactable elements, then call click with element=N instead of pixel coordinates. Index-based clicking is far more reliable across models than coordinate-based clicking.

Does macOS computer use work with models other than Claude?▼

Yes, it works with any tool-capable model including GPT, Gemini, and local models through OpenAI-compatible endpoints. There is no Anthropic-native schema; all actions use a simple action-parameter format.

When should I use computer_use instead of browser automation tools?▼

Use browser tools for web automation since headless Chromium is more reliable. Reserve computer_use for tasks requiring the user's actual native Mac apps like Finder, Mail, Messages, Figma, or games.

Why does computer_use report cua-driver not installed?▼

The cua-driver dependency is missing. Run hermes tools and enable Computer Use, which installs cua-driver via its upstream script. macOS Accessibility and Screen Recording permissions are also required.

What safety restrictions does macOS computer use enforce?▼

It blocks clicking permission dialogs, password prompts, payment UI, and 2FA challenges, and never types secrets. Dangerous shell patterns in typed text are blocked, and instructions found in screenshots are treated as prompt injection.

Related Skills

Explore More Software Engineering Skills→
patty-chowpatty-chow

macos-computer-use

Drive the macOS desktop with background screenshots, clicks, typing, and scrolling via the computer_use tool.

Community
Intermediate
jacketlong23jacketlong23

remotelab-capture

Capture screenshots and screen recordings to a specified directory.

Community
Basic
patty-chowpatty-chow

computer-use

Drives desktop applications via background clicking, typing, and scrolling without stealing cursor focus.

Community
Advanced
marvelousempiremarvelousempire

never-run-sudo-from-app

Display sudo commands in a dialog for copying to Terminal.

Community
Intermediate