What problem does it solve? Automating tasks in native desktop applications normally requires stealing the user's cursor and keyboard focus, interrupting their work. This Skill drives the desktop in the background — clicking, typing, scrolling, and dragging — without moving the real cursor, switching virtual desktops, or raising windows, and it works with any tool-capable model. ## Core Features & Use Cases - Element-index targeting: Capture screenshots with numbered overlays and an accessibility-tree index, then click elements by index instead of unreliable pixel coordinates. - Verify-and-escalate ladder: Every input action returns a structured verdict (confirmed, unverifiable, suspected_noop) with recommended escalation from background input to pixel coordinates, typed browser-page routes, or foreground delivery. - Cross-platform support: Works on macOS (AX), Windows (UIA), and Linux (AT-SPI), with platform-specific keyboard shortcuts and failure diagnostics via hermes computer-use doctor. - Use Case: Ask the agent to fill out a form in a native desktop app like Figma or Mail while you keep typing in your editor — the agent captures the app, clicks fields by element index, types values, and verifies each step without ever taking your focus. ## Quick Start Ask the agent to open a specific desktop application and complete a task in it, such as clicking through a dialog or filling a form, while you continue working in another window.
