Qt Pilot

Qt Pilot provides a programmable MCP server that lets you run Qt and PySide6 applications headlessly, capture UI state, and simulate user interactions. With this you can automate visual testing, perform widget discovery, and monitor app health in a headless environment using an MCP client.

How to use

You interact with the Qt Pilot MCP Server through an MCP client. Start by launching a Qt application in a headless display, then discover interactive widgets, simulate clicks, hovers, and keyboard input, and finally verify visuals by capturing screenshots or querying widget state. Use the provided tools to drive your app from your test scripts or AI-assisted workflows.

Typical usage patterns include launching an app, listing available widgets, performing actions, waiting for the UI to settle, and capturing results. For example, you might start the app in a test harness, locate a login button by its object name, click it, type into input fields, and take a screenshot to confirm the outcome.

Key capabilities you can rely on include: launching Qt apps headlessly, capturing screenshots for visual verification, simulating clicks and keyboard input, discovering widgets by object name, monitoring app health via stderr, and obtaining full Qt introspection data through Qt APIs.

How to install

Prerequisites you need before installing are Python 3, Git, and a headless display tool (Xvfb). You also need the required Python packages for the MCP server.

# 1) Clone the Qt Pilot plugin into your Claude plugins directory
git clone https://github.com/neatobandit0/qt-pilot.git ~/.claude/plugins/qt-pilot

# 2) Install Python dependencies
pip install -r ~/.claude/plugins/qt-pilot/requirements.txt

# 3) Alternative manual installation (copy the plugin directly)
cp -r qt-pilot ~/.claude/plugins/

# 4) Configure your MCP server entry (example)
{
  "mcpServers": {
    "qt-pilot": {
      "type": "stdio",
      "command": "python3",
      "args": ["/path/to/qt-pilot/server/main.py"]
    }
  }
}

Install the runtime dependencies for Qt Pilot and headless operation. You will also need Xvfb to provide a virtual display for Qt GUI testing.

# Install required Python packages
pip install mcp PySide6

# Install Xvfb for headless display
# Debian/Ubuntu
sudo apt install xvfb

# RHEL/CentOS/Fedora
sudo yum install xorg-x11-server-Xvfb

# macOS (via Homebrew)
brew install xquartz

Additional configuration and notes

To enable the MCP server, you typically point your client to the stdio-based server entry shown above. The server runs the Qt application inside a virtual display and exposes a set of tools to interact with the UI.

If you need a quick start, ensure your Qt application shows at least one window and that all interactable widgets have object names set. This enables widget discovery and reliable interaction via the MCP tools.

Troubleshooting

Widget discovery issues usually mean widgets lack an object name. Use the discovery tool to list available widget names and set object names in your Qt code as needed.

If the application is not running, verify you started the app through the MCP client and that the path to your script or module is correct.

If the app crashes or produces errors, inspect the stderr stream captured by the server for crash messages or stack traces.

Screenshots may be blank if the window was not rendered before capture. Wait for idle or rendering to complete before capturing visuals.

Tools and capabilities overview

The MCP tools available with Qt Pilot let you perform a sequence of operations on the running application, such as launching an app, discovering widgets, simulating user actions, querying widget information, waiting for idle, and closing the app. Use these tools to build automated test flows or AI-assisted UI verifications.

Security and environment considerations

Run the MCP server inside a controlled environment with restricted access to test artifacts and application binaries. Avoid leaking sensitive data through widget text or stdout/stderr. Use proper isolation for each test run to prevent cross-test contamination.

Notes

The server relies on a headless display via Xvfb and requires PySide6 for Qt interaction. Ensure your test layout adheres to the requirements for widget discovery and that your application exposes the necessary object names for reliable automation.

Available tools

launch_app

Launch a Qt application headlessly, either by script or module, establishing the test harness within the headless environment.

capture_screenshot

Capture the current window as an image for visual verification.

click_widget

Click a widget by its object name, selecting which mouse button to use.

hover_widget

Hover the cursor over a specified widget to trigger hover effects.

type_text

Type text into a specified widget or into the currently focused widget.

press_key

Simulate a key press with optional modifiers to drive keyboard input.

find_widgets

List widgets that match a name pattern to locate controls for interaction.

get_widget_info

Retrieve detailed information about a widget, such as type, visibility, size, and state.

get_app_status

Check if the application is running and obtain diagnostics including display and stderr.

wait_for_idle

Wait for the Qt event queue to settle after actions to ensure end results are ready.

close_app

Close the running Qt application and clean up resources.