What problem does it solve? Driving a user's real desktop applications normally requires fragile pixel-coordinate automation that steals the cursor and keyboard focus. This Skill lets an AI agent click, type, scroll, and drag inside native apps in the background, using accessibility-tree element indices instead of raw coordinates, without interrupting the user's own work. ## Core Features & Use Cases - Element-index targeting: Capture a screenshot with numbered overlays on every interactable element (SOM mode), then click or type by element index rather than pixel coordinates, which works reliably with any tool-capable model. - Background operation: Actions never move the user's cursor, steal keyboard focus, raise windows, or switch virtual desktops, so the user can keep working in another window. - Cross-platform support: Works on macOS (AX), Windows (UIA), and Linux (AT-SPI) through the cua-driver layer, with platform-appropriate keyboard shortcuts and failure diagnostics. - Use Case: Ask the agent to fill out a form in a native desktop app like Mail or Figma while you continue typing in your editor; the agent captures the app, clicks fields by index, and verifies each step with follow-up screenshots. ## Quick Start Ask the agent to open a specific desktop application and perform a GUI task, such as "capture Chrome and click the Sign In button", and it will screenshot the app, index its elements, and act on them in the background.
