win-ui-auto_skill

Overview

This skill provides Windows UI automation using built-in PowerShell, .NET UIAutomation, and Win32 APIs to inspect elements, move the mouse, send keystrokes, and manage windows and processes. It requires no external dependencies and works on Windows 10/11 with PowerShell 5.1+. Use it to drive GUI workflows, capture element positions, and integrate with screenshot tools for visual verification.

How this skill works

The skill enumerates the UI Automation tree to discover windows and controls, returning control type, name, automation ID, and bounding rectangles. It performs interactions by moving the cursor (SetCursorPos), generating mouse events (mouse_event), sending keyboard input (SendKeys), and calling Win32 functions to focus, move, resize, minimize, maximize, or close windows. Scripts are executed through powershell.exe commands so you can embed commands in automation pipelines or call them from other tools.

When to use it

• Automating repetitive GUI tasks when an API is unavailable.
• Locating controls by accessibility properties rather than image matching.
• Filling forms, triggering menu actions, or sending hotkeys programmatically.
• Combining with screenshots for visual validation and layout analysis.
• Managing window placement, focus, and lifecycle during test runs.

Best practices

• Always list and focus the target window before sending keystrokes (SendKeys requires focus).
• Use the See UI Elements command to get element center coordinates for reliable clicks.
• Prefer UI Automation properties (Name, AutomationId, ControlType) over fixed coordinates when possible.
• Run scripts with the same privilege level as the target app; elevated apps may ignore non-elevated input.
• Limit returned elements (truncate at 100) during exploration to keep output manageable.

Example use cases

• Open Notepad, focus its window, click the edit control, type multi-line text, and save with Ctrl+S.
• Enumerate buttons in an app window and click a specific Button by its center coordinates.
• Automate a workflow: list windows → focus target → take screenshot → analyze layout → click/type.
• Move and resize an application window for consistent test layouts before running UI checks.
• Trigger keyboard shortcuts (Ctrl+C, Ctrl+V, Alt+F4) as part of scripted operations.