skills.rest
patty-chow
patty-chowCOMMUNITY
Explore all skills by this author →
GitHub

cua-driver

Automates native macOS apps via accessibility-tree snapshots and element-indexed clicks without stealing focus.

Updated May 28, 2026
Search Tags:#macos#gui automation#accessibility tree#cua-driver#background automation#element index#applescript alternative
One-click install
npx skills add https://github.com/patty-chow/the-stable --skill cua-driver-patty-chow
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: cua-driver
Source: https://github.com/patty-chow/the-stable/tree/main/skills/openclaw-imports/cua-driver
Command: npx skills add https://github.com/patty-chow/the-stable --skill cua-driver-patty-chow

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Driving macOS GUI applications programmatically usually means focus-stealing AppleScript, cursor-warping click tools, or fragile pixel coordinates. This Skill teaches an agent to operate native macOS apps entirely in the background through the cua-driver CLI, so the user's frontmost app, cursor, and Space never change. ## Core Features & Use Cases - Snapshot-before-and-after loop: Every action is bracketed by get_window_state calls that return the accessibility tree plus a screenshot, so the agent verifies each click, keystroke, or value write actually landed. - Backgrounded dispatch: Clicks, typing, scrolling, and hotkeys are delivered per-pid via SkyLight/CGEvent paths, with a launch_app primitive that starts apps hidden and a FocusRestoreGuard that blocks activation. - Web-app and edge-case coverage: Companion references document Chromium/Electron/WebKit quirks, minimized-window keyboard-commit workarounds, tabs-vs-windows strategy, and optional trajectory recording and replay for regression testing. - Use Case: Ask the agent to compute something in Calculator, navigate Finder to a folder, or open a URL in Chrome while you keep typing in your editor — the target app never comes to the foreground. ## Quick Start Ask the agent to open the Downloads folder in Finder or click a specific button in any macOS app, and it will follow the launch_app, snapshot, act, verify loop automatically.

Frequently Asked Questions about cua-driver

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate a macOS app without stealing focus?▼

Use cua-driver's launch_app to start the app hidden, then call get_window_state to snapshot its accessibility tree and dispatch actions by element_index. All input is delivered per-pid via SkyLight, so the frontmost app and cursor never change.

How do I click a button in a macOS app using the accessibility tree?▼

Call get_window_state with the app's pid and window_id to get a tree where every actionable element is tagged with an index, then call click with that element_index. Re-snapshot afterward to confirm the action landed, since indices are invalidated on every new snapshot.

Why does cua-driver return 'No cached AX state' or 'Invalid element_index'?▼

Element indices are cached per (pid, window_id) pair and replaced on every snapshot, so indices from a previous turn or a different window do not resolve. Call get_window_state again in the same turn with the same window_id before acting.

Does cua-driver work with Chrome, Electron, and other web-based apps?▼

Yes, but Chromium and Electron apps ship with their web accessibility tree disabled; cua-driver enables it on the first snapshot, which may need one retry. The WEB_APPS.md reference covers URL navigation via launch_app, minimized-window keyboard-commit workarounds, and scroll handling.

Why doesn't pressing Return work on a minimized Chrome window?▼

On minimized windows the keystroke reaches the app but renderer focus is never established, so commit keys like Return silently no-op. Use set_value to write the field's value directly, or AX-click a commit-equivalent button such as Go or Submit.

What permissions does cua-driver need on macOS?▼

CuaDriver.app requires Accessibility and Screen Recording grants in System Settings, verifiable with cua-driver check_permissions. It also requires macOS 14 or newer because it depends on SkyLight private APIs stabilized in Sonoma.

Related Skills

Explore More Software Engineering Skills→
islee23520islee23520
5

cua-driver

Drive native macOS apps via accessibility tree, mouse, keyboard, and screenshots.

Community
Advanced
NVIDIANVIDIA
3.2k

i4h-workflow-dataset-mimic

Expand HDF5 robot demonstration recordings by cloning trajectories with action and state noise.

Official
Intermediate
thistleknotthistleknot

cua-desktop-agent

Automate desktop GUI tasks via vision-language perception and CUA-formatted actions.

Community
Advanced
autonomy-cloudautonomy-cloud
2

macos-automation

Automate native macOS app interactions using AppleScript via osascript.

Official
Intermediate