skills.rest
polyuiislab
polyuiislabCOMMUNITY
Explore all skills by this author →
GitHub

mac-use

Automate macOS GUI interactions using OCR-based element detection and canvas coordinates.

1.2k|53|Updated Dec 24, 2025
Search Tags:#automation#macos#scripting#window-management#gui-automation#vision-ocr
One-click install
npx skills add https://github.com/polyuiislab/infiAgent --skill mac-use
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: mac-use
Source: https://github.com/polyuiislab/infiAgent/tree/main/skills/mac-use
Command: npx skills add https://github.com/polyuiislab/infiAgent --skill mac-use

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pyobjc-framework-Vision, pyautogui, Pillow, and includes scripts (resource) components.

What problem does it solve?

Automate repetitive GUI interactions on macOS by recognizing on-screen text and providing a reliable, element-based way to interact with apps.

Core Features & Use Cases

  • OCR-based text detection and element identification in macOS apps
  • Numbered element clicking, typing, scrolling, and key presses across macOS windows
  • Robust window activation, coordinate mapping, and fallback canvas coordinates for unlabeled icons
  • Use cases include automating routine UI tasks, data entry, and GUI testing on macOS

Quick Start

Open a macOS app, run the screenshot command to detect elements, then click a numbered element or type text.

Frequently Asked Questions about mac-use

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate macOS GUI interactions without accessibility identifiers?▼

You can automate macOS GUI interactions using OCR-based element picking. The Skill detects on-screen text via Vision OCR, maps numbered elements on a 1000x1000 canvas, and executes scriptable clicks and typing for apps lacking accessibility labels.

Can I click buttons and type text in desktop apps using Python scripting?▼

Yes, you can click numbered elements, type text, scroll, and press keys in desktop apps. Through a dedicated scripts directory, it applies PyAutoGUI and Vision OCR to interact with macOS windows after taking a screenshot to detect UI elements.

Does this OCR-driven automation work for unlabeled icons in macOS apps?▼

OCR-driven automation handles unlabeled icons using a fallback 1000x1000 canvas coordinate system. When Vision OCR cannot detect text on an icon, you can navigate and interact using mapped canvas coordinates instead of text labels.

What do I need to set up to start automating macOS windows?▼

You need a macOS environment with the required dependencies installed, including pyobjc-framework-Vision, pyautogui, and Pillow. Open a target macOS app, run the screenshot command to detect elements, then execute clicks or typing.

What are the limitations of using Vision OCR for GUI testing on macOS?▼

Vision OCR for GUI testing relies on on-screen text recognition, meaning purely graphical elements without text require fallback canvas coordinates. It is best suited for routine UI tasks, data entry, and testing rather than pixel-perfect image recognition.

Related Skills

Explore More Software Engineering Skills→
iBz-04iBz-04
63

mac-use

Control macOS GUI apps by clicking OCR-detected numbered elements from screenshots.

Community
Intermediate
ThejanaJayalathThejanaJayalath

e2e-testing

Standardizes Playwright end-to-end testing with Page Object Model and CI/CD integration.

Community
Advanced
EdanStarfireEdanStarfire
15

process-manager

Manage processes by PID across Windows, Linux, and macOS.

Community
Basic
sky770825sky770825

screen-vision

Extract screen text and locate element coordinates using macOS Vision Framework.

Community
Intermediate