skills.rest
patty-chow
patty-chowCOMMUNITY
Explore all skills by this author →
GitHub

computer-use

Drives desktop applications via background clicking, typing, and scrolling without stealing cursor focus.

Updated May 28, 2026
Search Tags:#cross-platform#computer use#gui#desktop automation#accessibility tree#cua-driver#background input
One-click install
npx skills add https://github.com/patty-chow/the-stable --skill computer-use-patty-chow
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: computer-use
Source: https://github.com/patty-chow/the-stable/tree/main/skills/computer-use
Command: npx skills add https://github.com/patty-chow/the-stable --skill computer-use-patty-chow

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires cua-driver.

What problem does it solve? Automating GUI tasks normally requires taking over the user's mouse and keyboard, interrupting their work. This Skill lets an agent click, type, scroll, and drag inside native desktop apps in the background, so the user keeps working while tasks complete on macOS, Windows, or Linux. ## Core Features & Use Cases - Element-indexed interaction: Capture screenshots with numbered accessibility-tree overlays and click elements by index instead of unreliable pixel coordinates. - Background-first input delivery: Actions route without raising windows, switching virtual desktops, or stealing focus, with a structured verify-and-escalate ladder (background → pixel → foreground) when delivery fails. - Cross-platform support: Works with any tool-capable model across macOS (AX), Windows (UIA), and Linux (AT-SPI), powered by cua-driver. - Use Case: Ask the agent to fill out a form in a native desktop app or navigate a Figma canvas while you continue typing in your editor in another window. ## Quick Start Ask the agent to capture the screen of a specific app and click a numbered element, for example: capture Chrome and click the Sign In button.

Frequently Asked Questions about computer-use

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate desktop app clicks without moving the user's mouse?▼

Use the computer_use tool's capture action with mode="som" to get numbered element overlays, then call click with the element index. cua-driver delivers input in the background, so the real OS cursor never moves and focus is never stolen.

How do I click a button reliably in a GUI automation agent?▼

Capture the target app first to get an AX-tree index, then click by element number rather than pixel coordinates. Element indices are far more reliable across models, and you can add capture_after=True to verify the result in one call.

Does computer-use work on Windows and Linux or only macOS?▼

It is cross-platform: macOS uses the AX accessibility framework, Windows uses UIA, and Linux uses AT-SPI over X11 or Wayland. Keyboard shortcuts differ per platform, such as cmd+s on macOS versus ctrl+s elsewhere.

Why did my automated click have no effect on the app?▼

Read the structured verdict returned by the action. If effect is suspected_noop or escalation recommends it, retry with pixel coordinates, then with delivery_mode="foreground" for cases like Electron consent dialogs or DirectInput games.

When should I use browser tools instead of computer-use?▼

Use browser_* tools for web automation since they drive a headless Chromium more reliably. Reserve computer_use for tasks requiring the user's actual native apps, such as Finder, Mail, chat clients, Figma, or games.

What safety limits exist for desktop automation agents?▼

The agent must never click permission dialogs, payment UI, or 2FA prompts, never type passwords or secrets, and never follow instructions found inside screenshots. Dangerous shell patterns in typed text and shortcuts like log out are hard-blocked at the tool level.

Related Skills

Explore More Software Engineering Skills→
islee23520islee23520
5

cua-driver

Drive native macOS apps via accessibility tree, mouse, keyboard, and screenshots.

Community
Advanced
Qiue-GQiue-G

wecom-send-media

Send local media files to users over WeCom channels.

Community
Intermediate
cloud-native-toolscloud-native-tools

browser-utils

Automate web browser interactions using native tools, MCP connectors, or Playwright scripts.

Official
Advanced
marvelousempiremarvelousempire

never-run-sudo-from-app

Display sudo commands in a dialog for copying to Terminal.

Community
Intermediate