midscene-computer-browser_skill

Overview

This skill provides vision-driven browser automation using Midscene and Puppeteer. It operates entirely from screenshots, so no DOM access or accessibility labels are required, and it can interact with any visible element regardless of technology stack. Each target URL opens in a new browser tab and the headless Chrome instance persists across commands for multi-step workflows.

How this skill works

The skill launches a Puppeteer-backed headless Chrome via the Midscene CLI and maps each CLI call to a screenshot-analyze-act loop. You connect to a URL, take a screenshot to observe the page state, then issue high-level act commands that describe desired UI effects (click, type, select, scroll). All midscene commands must run synchronously and one at a time so the agent can read screenshots before choosing the next action.

When to use it

• Browse, navigate, or open web pages programmatically
• Scrape or extract data from pages when DOM access is unavailable
• Fill forms, click buttons, and interact with visual UI elements
• Validate or test frontend UI behavior and visual flows
• Take screenshots or record step-by-step page state for reports

Best practices

• Always run connect --url first to open the target tab before other commands
• Take a screenshot after navigation or actions that change the page to confirm readiness
• Describe UI targets in natural, specific visual terms (e.g., 'blue Submit button in contact form')
• Run only one midscene command at a time and allow enough time for completion
• Batch related UI steps into a single act command to reduce round-trips
• Close the browser when finished to free resources and summarize generated files

Example use cases

• Log into a web app by filling email/password and clicking Login, then capture a post-login screenshot
• Scrape product listings by scrolling a page and extracting visible text and images
• Automate a multi-step checkout flow by selecting options, filling forms, and confirming orders
• Run visual regression checks by capturing screenshots before and after UI changes
• Collect screenshots and logs for manual review or QA reporting