OmniParser Autogui

Provides screen analysis via OmniParser and automatic GUI interactions for Windows, enabling automated GUI tasks.

other

59

GitHub Stars

other

Language

6 months ago

First Indexed

2 months ago

Catalog Refreshed

Documentation & install

Readme and setup notes from the catalogue, plus a client-ready config you can copy for your MCP host.

Installation

Add the following to your MCP client configuration file.

Configuration

View docs

{
  "mcpServers": {
    "non906-omniparser-autogui-mcp": {
      "command": "uv",
      "args": [
        "--directory",
        "D:\\\\CLONED_PATH\\\\omniparser-autogui-mcp",
        "run",
        "omniparser-autogui-mcp"
      ],
      "env": {
        "OCR_LANG": "en",
        "PYTHONIOENCODING": "utf-8"
      }
    }
  }
}

This MCP server uses OmniParser to analyze on-screen content and automatically drive GUI interactions, enabling automated tasks on Windows. It runs locally and integrates with your MCP client to provide seamless screen parsing and control.

How to use

You will run this MCP server locally and connect to it from your MCP client. The server analyzes the screen with OmniParser and then performs GUI actions automatically based on what it parses. Use it when you want to automate repetitive GUI tasks on Windows.

How to install

Prerequisites: You need Python available on your system and the UV runtime installed to manage MCP servers.

Clone the project, install dependencies, and set up models. Run the commands below in your terminal.

git clone --recursive https://github.com/NON906/omniparser-autogui-mcp.git
cd omniparser-autogui-mcp
uv sync
set OCR_LANG=en
uv run download_models.py

Configuration and usage notes

After cloning, add the MCP server configuration to your Claude desktop config. This enables your client to start and connect to the local MCP server using UV.

claude_desktop_config.json
{
  "mcpServers": {
    "omni_autogui_mcp": {
      "command": "uv",
      "args": [
        "--directory",
        "D:\\CLONED_PATH\\omniparser-autogui-mcp",
        "run",
        "omniparser-autogui-mcp"
      ],
      "env": {
        "PYTHONIOENCODING": "utf-8",
        "OCR_LANG": "en"
      }
    }
  }
}

Additional configuration options

The MCP supports additional environment variables for customizing OmniParser and server behavior. The following are the key options you can set in your environment block or per-server config.

OMNI_PARSER_BACKEND_LOAD: set to 1 if you encounter issues with other clients.
TARGET_WINDOW_NAME: specify a window name to limit operations to a particular window; omit to operate on the full screen.
OMNI_PARSER_SERVER: point OmniParser processing to a remote server, e.g., 127.0.0.1:8000, and start the server with uv run omniparserserver.
SSE_HOST / SSE_PORT: enable communication over SSE instead of stdio if needed.
SOM_MODEL_PATH, CAPTION_MODEL_NAME, CAPTION_MODEL_PATH, OMNI_PARSER_DEVICE, BOX_TRESHOLD: for advanced OmniParser configurations. These are usually not necessary for standard setups.

Available tools

OmniParser GUI Automation

Parses screen content with OmniParser and automatically drives GUI actions to automate tasks on Windows.