web-infra-dev/midscene-skills
Overview
This skill enables vision-driven iOS device automation using the Midscene CLI. It operates entirely from screenshots, so no DOM or accessibility labels are required. Use natural language prompts to control iPhone and iPad screens via WebDriverAgent and Midscene.js.
How this skill works
The agent connects to a device, captures screenshots, and uses a configured visual model to interpret the UI. Actions are executed through the Midscene CLI (npx @midscene/ios@1) using synchronous commands: connect, take_screenshot, act, and disconnect. The act command accepts high-level, goal-oriented prompts and performs taps, swipes, typing, and navigation based on on-screen visuals.
When to use it
- Testing or automating iOS apps that lack accessibility labels or stable DOM.
- Interacting with hybrid or custom-rendered UI elements that are only visible on screen.
- Performing exploratory manual flows where screenshots guide decision-making.
- Automating UI tasks on physical iPhones and iPads via WebDriverAgent.
- Running device-driven scenarios that require visual context rather than element trees.
Best practices
- Run every midscene command synchronously and wait for its output before the next step.
- Execute only one midscene command at a time; do not chain commands in the background.
- Batch related steps into a single act prompt (describe the full goal and desired effect).
- Be specific about UI targets and locations (e.g., 'top-right Settings icon' or 'third list item').
- Allow extra time for commands to complete—complex actions can take a minute or more.
- Collect and summarize screenshots, logs, and report files at the end of a session.
Example use cases
- Open Settings, navigate to Wi‑Fi, and report the connected network name.
- Tap a Delete button and confirm the resulting alert dialog, then capture the final screen.
- Fill a login form (username and password) and submit, returning a screenshot of the post-login state.
- Swipe through an onboarding carousel and take screenshots at each step for QA.
- Automate navigation and data extraction from an app with custom-rendered UI elements.
FAQ
You must set MIDSCENE_MODEL_API_KEY, MIDSCENE_MODEL_NAME, MIDSCENE_MODEL_BASE_URL, and MIDSCENE_MODEL_FAMILY either in your system environment or a .env file.
Why must commands run synchronously?
Midscene relies on an analyze-act loop where each screenshot informs the next action; background or concurrent commands break that loop and produce unreliable results.
2 skills
This skill automates iOS interactions using vision and screenshots, enabling reliable device control from natural language prompts.
This skill enables hands-free browser automation using natural language to navigate, extract data, and interact with web pages via Midscene CLI.
