midscene-automation_skill

Overview

This skill automates web browser interactions via the Midscene CLI using plain natural language. It controls a persistent Chrome instance to navigate sites, click and type, extract structured data, and capture screenshots. The browser session preserves cookies and state between commands for multi-step workflows.

How this skill works

You send simple commands to the CLI (navigate, act, query, assert, screenshot, close) and Midscene interprets them using AI-powered visual and semantic understanding of the page. The tool locates elements without CSS selectors by reasoning about labels, visuals, and context. Commands return JSON with success status, messages, extracted data, and screenshot paths for verification.

When to use it

• Open or navigate to a specific website or page
• Automate form filling, button clicks, scrolling, or other UI interactions
• Extract structured data (product lists, headlines, prices) from a page
• Take full-page screenshots for verification or documentation
• Run validation checks/assertions during testing or monitoring

Best practices

• Always run the navigate command before performing interactions
• Be specific in natural language (e.g., "the blue Submit button in the login form")
• After each command, review the returned screenshot to confirm state and troubleshoot failures
• Check setup.json and complete setup steps before running any browser commands
• Close the browser session with the close command when finished to free resources

Example use cases

• Multi-step login: navigate to login page, type credentials, click submit, assert login success, screenshot
• Web scraping: navigate to a products page and query "extract product names and prices in JSON"
• Form automation: navigate to signup, act to fill fields and click Register, then assert confirmation text
• Visual QA: navigate to a page, take screenshots of different states, and compare outputs
• Search workflow: navigate to search engine, act to enter query and press Enter, then query top result titles