airpoint_skill

Overview

This skill lets you control a Mac through natural language using Airpoint’s AI computer-use agent. It opens apps, clicks UI elements, reads on-screen text, types, scrolls, and automates multi-step workflows from the command line. Everything runs via the airpoint CLI and returns a text summary plus optional screenshots for verification.

How this skill works

You send a natural-language instruction to the on-device agent with airpoint ask "<instruction>" and the agent perceives the screen (accessibility tree + screenshots + visual locator), plans actions, executes mouse/keyboard/window operations, and verifies results. The CLI blocks until the task finishes (up to five minutes) and outputs a concise report and paths to any screenshots. The agent can use a visual locator model (Gemini) for robust target-finding when configured, and supports stop, status, and settings commands for control and inspection.

When to use it

• Automating repetitive UI workflows that span multiple apps (e.g., export → upload → notify).
• Interacting with apps that lack APIs by instructing the agent to click, type, and navigate the UI.
• Quick tasks where typing a single natural-language instruction is faster than manual steps.
• Extracting on-screen text or summaries from emails, dialogs, or webpages.
• Verifying visual results with returned screenshots after a command runs.

Best practices

• Ensure the Airpoint macOS app is running and the CLI is installed and on PATH.
• Grant Accessibility and Screen Recording permissions in System Settings before use.
• Configure an AI model API key in the app; add a Gemini key for visual locator accuracy.
• Use clear, stepwise instructions and prefer single-goal commands to reduce ambiguity.
• Use --json when parsing output programmatically and airpoint stop if a task stalls.

Example use cases

• Open Safari, search for a term, and take a screenshot of the results page.
• Open Mail, find the latest message from a contact, and summarize it.
• Enable a system setting (e.g., Dark Mode) by navigating System Settings.
• Start Spotify and play a playlist, using --no-wait for fire-and-forget execution.
• Capture current screen state with airpoint see before running a multi-step task.