Native DevTools

You can empower AI agents to interact with native desktop and Android apps by running a local MCP server that provides eyes, hands, and window management. This server delivers screenshots, OCR, input simulation, and UI exploration for native applications, not just browsers, enabling automated tasks, testing, and RPA across macOS, Windows, and Android devices.

How to use

Install and run the MCP server using the commands below, then connect your AI agent to the running server. You can choose either a quick one-off run with npx or a permanent setup via a global install. The server exposes a set of actions like taking screenshots, clicking, typing, and finding text or UI elements to let your agent automate native apps and Android devices.

How to install

Prerequisites: you need Node.js (version 18 or newer) for the npm/npx options, and Rust toolchain if you plan to build from source.

# Option 1: Run with npx (no install needed)
npx -y native-devtools-mcp

# Option 2: Global install
npm install -g native-devtools-mcp

# Option 3: Build from source (Rust)
# Cloning and building from source
git clone https://github.com/sh3ll3x3c/native-devtools-mcp
cd native-devtools-mcp
cargo build --release
# Binary: ./target/release/native-devtools-mcp

# To include Android device support, enable the android feature flag
cargo build --release --features android

Configuration and start options

Two straightforward ways exist to run the MCP server from the source or using a prebuilt package. Choose the method that matches your workflow.

Option A: Run with npx (no install needed) is the simplest way to start immediately without building locally.

Option B: Build from source and run the resulting binary to have a locally compiled MCP server for your environment and features like Android support.

Two approaches to interaction

You can use the MCP server in two interaction modes depending on your needs:

1. Visual approach (Universal): Take screenshots, analyze visually, and click at coordinates. Use tools like take_screenshot, find_text, click, type_text, and, for non-text elements, load_image / find_image.

2. Structural approach (AppDebugKit): If an app exposes a UI tree via AppDebugKit, you can query and act on the UI structure with app_connect, app_query, and app_click.

Template matching (find_image)

Use find_image for non-text targets such as icons or custom controls when OCR or text search cannot identify the element.

Typical flow:

Take a screenshot of the target app -> load an image template -> find_image -> click at the matched coordinates

Android support

Android support is optional and lets the MCP server talk to Android devices over ADB for screenshots, input, UI element search, and app management.

Prerequisites: ADB must be installed, USB debugging enabled on the device, and the ADB server running.

Building with Android support requires enabling the android feature during the Rust build.

Typical workflow demonstrates listing devices, connecting, taking screenshots, locating text, and interacting with the device.

Architecture and how it works

The MCP server communicates with clients over JSON-RPC 2.0, performs system-level actions on the host, and optionally interfaces with AppDebugKit or Android devices.

Privacy, safety & best practices

Everything runs locally on your machine; no data is sent to external servers. When the agent is driving input (clicks and typing), avoid moving your mouse or typing to prevent conflicts with simulated inputs. Keep the target window visible to ensure correct focus and element targeting.

Required permissions (macOS)

For macOS, grant Screen Recording and Accessibility permissions to the host application so the MCP server can capture the screen and simulate input. Restart the host app after granting permissions.

Notes for Windows users

On Windows, the MCP server uses standard Win32 APIs for screen capture and input. It relies on UI Automation as the primary text-search mechanism, with OCR as a fallback. Running as Administrator may be required for certain privileged windows.

License

MIT © sh3ll3x3c

Available tools

take_screenshot

Capture the screen or a specific region and return an image with coordinates and OCR metadata.

click

Simulate a mouse click at given screen coordinates or relative to a window, enabling precise input automation.

type_text

Type text into the focused element or at specified coordinates to automate keyboard input.

find_text

Locate on-screen text using the platform accessibility API with OCR fallback, returning coordinates of the matched text.

load_image

Load a non-text template image for template matching to find icons or UI elements.

find_image

Perform template matching to locate a non-text UI element on the screen and return its coordinates.

app_connect

Connect to an AppDebugKit-enabled app for a structural UI tree interaction.

app_query

Query the UI tree of a connected app to discover elements and their properties.

app_click

Click a UI element by identifier in apps instrumented with AppDebugKit.

android_screenshot

Capture the Android device screen via ADB.

android_click

Tap the device screen at specified coordinates.

android_find_text

Find UI elements on Android by text using the device's UIAutomator tree.

android_type_text

Type text on the Android device.

android_launch_app

Launch an app on the connected Android device by package name.

android_list_devices

List ADB-connected devices to identify targets.

android_connect

Connect to a specific Android device by serial number.

android_disconnect

Disconnect from the current Android device.