tart-gui-automation

Automates GUI workflows in ephemeral Tart macOS VMs via VNC and tart exec.

1|Updated Nov 11, 2012
One-click install
npx skills add https://github.com/fairchild/dotfiles --skill tart-gui-automation-fairchild
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tart-gui-automation
Source: https://github.com/fairchild/dotfiles/tree/main/agents/shared/first-party-skills/tart-gui-automation
Command: npx skills add https://github.com/fairchild/dotfiles --skill tart-gui-automation-fairchild

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires paramiko, vncdotool, Pillow, pyyaml, and includes scripts (resource) and references (resource) components.

What problem does it solve? Verifying macOS GUI behavior requires a real desktop environment, but manual testing is slow and CI runners rarely offer interactive displays. This Skill lets an agent spin up disposable Tart VMs, drive the GUI over VNC, and capture screenshots as verification evidence without touching the host system. ## Core Features & Use Cases - Ephemeral VM lifecycle: Clone a base image, run headless by default, and tear down cleanly so no state leaks between runs. - Guest command execution: Run builds and shell commands via tart exec with automatic SSH fallback when the guest agent is unavailable. - VNC GUI interaction: Click, double-click, type, send keyboard shortcuts, scroll, drag, and run multi-step batches over a single VNC connection. - Target manifests and flows: Store calibrated UI landmarks in .tart/target.yaml and reusable workflow recipes in .tart/flows/ so coordinates are version-controlled with the app. - Use Case: After building a SwiftUI app in a macos-tahoe-xcode VM, click the toolbar menu landmark, capture a resized screenshot of the open dropdown, and tear down the VM as proof the workflow succeeded. ## Quick Start Ask the agent to start a Tart VM from the macos-tahoe-xcode base image, build and launch your app inside it, click through the main workflow by landmark, capture verification screenshots, and tear the VM down.

Frequently Asked Questions about tart-gui-automation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate macOS GUI testing in a Tart VM?▼

Start an ephemeral VM cloned from a base image, run build commands via tart exec, then drive the GUI over VNC with click, type-string, send-keys, and batch commands. Capture screenshots for evidence and run teardown to delete the clone.

How do I click UI elements without hard-coding pixel coordinates?▼

Store calibrated coordinates as named landmarks in a .tart/target.yaml manifest in your project. The harness resolves names via the --landmark flag on click and double-click, or inside batch steps, so recalibration is a YAML edit.

Which Tart image should I use for building Swift apps?▼

Use an xcode-tier image such as macos-tahoe-xcode, which includes Xcode, the Swift toolchain, and the guest agent. For screenshot-only automation, the smaller base tier is sufficient; vanilla images lack the guest agent and SSH.

Does tart exec work on all Tart macOS images?▼

No, tart exec requires the tart-guest-agent, which is preinstalled only in base, xcode, and runner tier images. On vanilla images the harness falls back to SSH after you enable Remote Login and run discover-ssh.

Why do my VNC clicks miss the target UI element?▼

The default framebuffer is 2048x1536 physical pixels, so logical point math underestimates y positions. Capture a screenshot, measure physical pixel coordinates, or run a calibration sweep and store results as landmarks.

Why does a second VNC connection fail in the same script?▼

vncdotool's Twisted reactor cannot restart within one process, so a second api.connect call raises ReactorNotRestartable. Use the harness batch command to run multiple operations over a single connection instead.