computer-use_skill

This skill enables headless desktop control on Linux servers by automating an XFCE/Xvfb environment with noVNC and xdotool.

Python

2.5k

GitHub Stars

2

Bundled Files

2 months ago

Catalog Refreshed

4 months ago

First Indexed

Readme & install

Copy the install command, review bundled files from the catalogue, and read any extended description pulled from the listing source.

Installation

Preview and clipboard use veilstrat where the catalogue uses aiagentskills.

npx veilstrat add skill openclaw/skills --skill computer-use

_meta.json812 B
SKILL.md6.1 KB

Overview

This skill provides full desktop GUI control for headless Linux servers by creating a virtual X11 display (Xvfb) running a minimal XFCE desktop. It exposes 17 scriptable actions (click, type, scroll, screenshot, drag, key hold, etc.) and includes VNC and noVNC for live viewing and debugging. Use it to automate desktop apps and browsers at the X11 level so automation is not detectable by websites.

How this skill works

The skill starts an Xvfb virtual display on :99 with a lightweight XFCE session and runs x11vnc + noVNC for remote viewing. Actions are implemented as small shell scripts that call xdotool, scrot and related tools to move the mouse, click, type text in chunks, press keys, scroll, drag, and capture screenshots. Most actions optionally return base64 PNG screenshots and follow a screenshot→analyze→act→verify workflow.

When to use it

Automating GUI-only applications on headless VPS/cloud instances
Running browser automation where webpage bot-detection must not see browser-level automation hooks
Testing GUI workflows that require a real desktop environment (menus, dialogs, file pickers)
Remote troubleshooting and live demos via VNC/noVNC
Batch interactions with legacy desktop apps that lack CLI or API controls

Best practices

Always export DISPLAY=:99 before invoking scripts
Start with a screenshot to determine coordinates, then act and re-screenshot to verify
Click to focus input fields before typing; long text is split into 50-char chunks with small delays
Use SSH tunnels for secure noVNC and VNC access (forward 6080 and 5900)
Prefer Chrome .deb on headless servers; avoid snap builds due to sandbox issues
Monitor systemd services (xvfb, xfce-minimal, x11vnc, novnc) and restart if needed

Example use cases

Automate form-filling and file uploads in a browser running on a headless server without browser automation flags showing
Take periodic full-desktop screenshots of a GUI process for monitoring or archival
Drive legacy desktop-only applications to perform batch conversion or export tasks
Interact with desktop installers or configuration dialogs during automated server provisioning
Provide a live desktop for remote debugging and manual intervention via VNC/noVNC

FAQ

SSH-tunnel ports 6080 and/or 5900 to the server then open http://localhost:6080/vnc.html or connect a VNC client to localhost:5900.

Why use this instead of browser-level automation tools?

This operates at the X11 level so websites cannot detect automation signals injected by browser automation APIs; it also supports non-browser desktop apps.