peekaboox_skill

Overview

This skill controls and automates an X11 Linux desktop GUI. It captures screenshots, locates and clicks UI elements, types text, sends hotkeys, scrolls, and manages windows (focus, move, resize, minimize, maximize, close). Use it for desktop automation, GUI testing, and remote control where X11 is available.

How this skill works

Tools capture the current X11 framebuffer and return image files or JSON metadata. You inspect screenshots (manually or with an image-capable model), determine pixel coordinates or window names, then invoke commands to click, type, send hotkeys, scroll, or change window state. All commands support a --json flag for machine-readable responses and require DISPLAY to point at an X11 session.

When to use it

• Automating repeatable GUI tasks on an X11 desktop
• End-to-end GUI testing for Linux applications running under X11
• Remote desktop control or scripted maintenance of graphical apps
• Filling forms, clicking buttons, or reproducing manual workflows
• Window management and layout automation for multi-app setups

Best practices

• Ensure DISPLAY is set (export DISPLAY=:0) and X11 session is active before running commands
• Capture a screenshot first to confirm current UI and coordinates before acting
• Prefer identifying targets by window name when possible to avoid brittle absolute coordinates
• Use --json output in scripts for reliable success/failure handling
• Disable compositor or adjust capture method if screenshots look incorrect

Example use cases

• Take a full-screen capture then click the Save button in a dialog using coordinates from the image
• Focus a Terminal window, type a command, and send Return to run it programmatically
• Maximize a browser window, interact with its UI, then resize and position it for a recording
• Run automated GUI tests: capture pre- and post-action screenshots and verify visual changes
• Script remote maintenance: open an app, apply settings via clicks, and close it