skills.rest
Logarn
LogarnCOMMUNITY
Explore all skills by this author →
GitHub

computer-use

Controls the macOS desktop through accessibility-tree-driven click, type, scroll, and AppleScript actions.

1|Updated Mar 30, 2026
Search Tags:#macos#applescript#desktop control#gui automation#accessibility tree#proxy tools#screen observation
One-click install
npx skills add https://github.com/Logarn/Worklin-ai --skill computer-use-logarn
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: computer-use
Source: https://github.com/Logarn/Worklin-ai/tree/main/assistant/src/config/bundled-skills/computer-use
Command: npx skills add https://github.com/Logarn/Worklin-ai --skill computer-use-logarn

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Desktop applications without a CLI or API cannot be automated through conventional scripting, forcing users to perform repetitive GUI interactions manually. This Skill lets the assistant observe the screen and operate macOS apps directly on the user's behalf. ## Core Features & Use Cases - Screen Observation: Captures the accessibility tree with element IDs and optional screenshots so actions target real UI elements instead of fragile pixel coordinates. - GUI Action Set: Provides click, double-click, right-click, text typing, keyboard shortcuts, scrolling, dragging, app switching, and AppleScript execution, all forwarded to a connected macOS client. - Use Case: Ask the assistant to open a design tool with no API, navigate its menus, rename a layer, and export a file, with each step executed through the connected desktop client. ## Quick Start Ask the assistant to open Safari on your Mac, navigate to a website, and click the sign-in button using the computer use tools.

Frequently Asked Questions about computer-use

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate macOS desktop apps that have no API?▼

Use the computer use tools to observe the screen's accessibility tree, then target UI elements by their element IDs for clicks, typing, scrolling, and drags. Actions are forwarded to a connected macOS client that performs them on the desktop.

How does the assistant know where to click on screen?▼

The computer_use_observe tool captures the accessibility tree, listing interactive elements with [ID] references like AXButton or AXTextField. Subsequent actions use these element IDs, which is more reliable than raw pixel coordinates, though x/y fallbacks exist.

Can I use AppleScript instead of clicking through the GUI?▼

Yes, computer_use_run_applescript executes AppleScript directly and is preferred over click/type because it does not move the cursor or interrupt foreground activity. The 'do shell script' command inside AppleScript is blocked for security.

Does computer use work without a connected desktop client?▼

No, all tools in this skill are proxy tools whose execution is forwarded to a connected macOS client via HostCuProxy. The skill is preactivated only for conversations with a connected desktop client and never executes actions locally.

When should I avoid using computer use automation?▼

Avoid it when a more specific skill or a CLI/API call can accomplish the task, such as gmail, calendar, contacts, or terminal-sessions. GUI automation is reserved for apps with no programmatic alternative.

Related Skills

Explore More Software Engineering Skills→
vellum-aivellum-ai
1.0k

computer-use

Control a macOS desktop via proxy tools to automate UI tasks.

Official
Intermediate
EpicenterHQEpicenterHQ
4.8k

codex-task-backlog-hygiene

Audit Codex tasks and archive completed chats while preserving unowned outcomes in BACKLOG.md.

Official
Intermediate
PotLockPotLock

gog

Manage Google Workspace components via the gog CLI with OAuth authentication.

Official
Advanced
understudy-aiunderstudy-ai
455

peekaboo

Automate macOS UI interactions and system tasks via command-line interface.

Official
Advanced