macpilot-ui-inspector_skill

Overview

This skill lets you inspect and interact with macOS UI elements using MacPilot accessibility APIs. It finds buttons, text fields, labels, and other controls by role, label, or screen position, then reads, clicks, or modifies them. The tool is designed for precise, scriptable automation and exploration of an app's accessibility tree.

How this skill works

The skill queries the macOS Accessibility (AX) APIs exposed by MacPilot to enumerate UI elements, inspect attributes, and perform actions like click, set-value, focus, and scroll. You can filter searches by role or label, request JSON output for reliable parsing, or locate elements by screen coordinates for fallback interactions. It also exposes commands to read keyboard shortcuts, dump the accessibility tree, and wait for elements to appear.

When to use it

• Automating a sequence that clicks buttons or fills form fields in a macOS app
• Diagnosing why a UI element is not reachable or reading its accessibility attributes
• Extracting the UI accessibility tree to discover element roles and hierarchy
• Running headless scripts that need reliable element identification via JSON output
• Clicking or interacting with elements found by screen coordinates as a fallback

Best practices

• Always call app focus before interacting to ensure the target app receives actions
• Use --json output when scripting so you can parse element properties reliably
• Apply --role filters to narrow searches and speed up discovery in complex apps
• Prefer label-based actions but fall back to elements-at coordinates if clicks fail
• Grant Accessibility permission to MacPilot in System Settings before use

Example use cases

• Find and click a 'Save' or 'Cancel' button inside TextEdit by label or role
• Read and set values in a search field in Safari, then submit the query
• Dump the full AX tree for an app to locate hidden menu items or controls
• Wait for a 'Download Complete' dialog and then click its confirm button automatically
• Locate elements at a screen coordinate and perform a fallback click when label-based actions fail