What problem does it solve? Automating the macOS GUI normally requires hijacking the user's cursor and keyboard focus, interrupting their work. This Skill lets an agent capture screens, click elements, type, scroll, and drag in the background — without moving the user's cursor, stealing focus, or switching Spaces — using any tool-capable model. ## Core Features & Use Cases - Element-index automation: Capture a screenshot with numbered overlays on every interactable element (SOM mode) plus an accessibility-tree index, then click by element number instead of unreliable pixel coordinates. - Full input control: Click, double-click, right-click, drag-and-drop, scroll, type Unicode text, and send keyboard shortcuts like cmd+s, all with optional inline post-action verification captures. - Safety guardrails: Hard rules against clicking permission dialogs, typing secrets, or obeying instructions found in screenshots, plus tool-level blocking of dangerous commands. - Use Case: Fill out a web form in Safari — capture the page to find field indices, click each AXTextField, type values, paste an SSH key from the clipboard, and submit, all while the user keeps working in another app. ## Quick Start Ask the agent to capture the Safari window with computer_use and click the sign-in button by its element index.
