xdotool-control_skill

Overview

This skill automates mouse, keyboard, and window operations on Linux desktops using xdotool and common utilities. It is designed to interact with GUIs when a browser CDP or programmatic API is unavailable, especially for clicking Chrome extension icons, switching tabs, typing into apps, and running screenshot-verify-click loops. The patterns are lightweight shell scripts and commands you can embed into larger automation workflows.

How this skill works

The skill issues xdotool commands to search for windows, activate them, move the mouse, click, type text, and send keypresses. It pairs xdotool with scrot for screenshots and optional ImageMagick compare for template matching in verify-and-click loops. Scripts follow safe sequences: find window → activate --sync → brief sleep → action (mousemove/click/type) and optional verification via screenshots.

When to use it

• Click a Chrome extension icon when extension UI has no programmatic API
• Type input into a GUI app or terminal from a script
• Switch or open browser tabs from a desktop automation flow
• Automate desktop UI tasks without a browser relay or remote protocol
• Run screenshot then verify then click loops for visually locating elements

Best practices

• Always windowactivate --sync before sending mouse or keyboard events to ensure focus
• Insert a short sleep (e.g., 0.2–0.5s) after activation to avoid race conditions
• Capture a toolbar or window snapshot first to validate coordinates visually before clicking
• Use xdotool type for text and xdotool key for special keys (ctrl+t, Return)
• Prefer screen-relative coordinates derived from getdisplaygeometry and getwindowgeometry rather than hardcoded values
• Install scrot and optionally ImageMagick for screenshotting and template matching

Example use cases

• Scripted click on a pinned Chrome extension icon to open its popup and extract data
• Automated tab navigation and URL entry for headless-like workflows where CDP is unavailable
• Typing commands into an active terminal window as part of a deployment or monitoring routine
• Screenshot-verify-click loop to find an unpinned extension icon image and then click it
• Resize, move, and raise windows before running a visual test or taking consistent screenshots