ui-element-ops_skill

Overview

This skill parses UI screenshots into structured element JSON and can operate a desktop UI using those parsed elements. It detects element type, bounding boxes (pixel and normalized), OCR text when available, and a clickable flag. It also supports overlays, coordinate calibration for multi-display/DPI/window offsets, and basic desktop actions like click, type, and hotkeys.

How this skill works

The parser inspects one or more screenshots and produces a machine-readable JSON schema with element id, normalized type, bbox_px, bbox_norm, text (OCR), and clickable. An optional overlay PNG visualizes labeled boxes. A companion operate module uses the parsed JSON to locate elements, wait for appearance/disappearance, click by id or coordinates, type text, press keys or hotkeys, take screenshots, and calibrate coordinates between parsed and actual displays.

When to use it

• You need structured coordinates and text for UI automation or testing from screenshots.
• You want to find UI elements by type or partial text and perform actions (click/type/wait).
• You must calibrate coordinates across multiple displays, DPI settings, or window offsets.
• You need an overlay visualization to verify detection coverage and bounding boxes.
• You want to capture a screenshot and parse it in one step for quick inspection.

Best practices

• Run the bootstrap once per machine to install weights and dependencies before parsing.
• Reuse recent elements.json for repeated automation rather than parsing in tight loops.
• Check output summary counts (total, clickable, by_type) to validate detection quality.
• Call out tiny text or dense icon layouts as potential OCR or type-class errors.
• Use calibrate when actual screen size or DPI differs from parsed image dimensions.

Example use cases

• Locate a 'Login' button by text-contains and click it as part of a login automation flow.
• Wait for a 'Continue' text to appear, then capture a screenshot and proceed.
• Calibrate parsed coordinates from a remote screenshot to a local multi-monitor setup.
• Generate an overlay.png to review detected buttons, inputs, and labels for QA.
• List all clickable elements and produce a summary of counts for test reporting.