Swift MCP GUI Server

You can run a Swift-based MCP Server that lets you control macOS programmatically via SwiftAutoGUI. This server exposes tools to move the mouse, click, type keys, scroll, capture the screen, read pixel colors, and run AppleScript, enabling automated workflows and remote control through MCP clients.

How to use

You connect to this server with an MCP client and call the available tools to perform actions on your macOS desktop. You can move the mouse to precise coordinates, click the left or right button, send keyboard shortcuts, scroll in a direction, capture screenshots, read pixel colors, or run AppleScript commands. Use Capture Screen or Capture Region to obtain images for processing, and Save Screenshot to store results locally. This setup is designed to work in automated environments while requiring full accessibility permissions for control.

How to install

Prerequisites you need before installing: macOS 15.0 or later, Swift 6.0 or later, and Xcode 16.0 or later.

# 1) Install the MCP server tools (example path shown for Swift-based tool)
swift package experimental-install

# 2) Add the MCP server command to your client configuration as shown in the example
# (the exact path may vary based on your setup)

Additional setup and configuration

Configure your MCP client to load the Swift MCP GUI server. The client should reference the Swift MCP GUI executable so it can receive and handle MCP requests.

Security considerations

This server requires full accessibility permissions in macOS to control the mouse and keyboard. Exercise caution when enabling remote connections and only connect trusted MCP clients.

Notes

The server provides a rich set of tools for GUI automation on macOS, including mouse, keyboard, screen capture, and AppleScript execution. Ensure you understand the security implications and manage access to prevent unauthorized control.

Available tools

moveMouse

Moves the mouse cursor to specified x and y coordinates. Inputs: x (number), y (number). Accepts integers, doubles, or string representations.

mouseClick

Performs a mouse click at the current cursor position. Input: button can be "left" or "right".

sendKeys

Sends keyboard keys or combinations. Input: keys is an array of strings representing key names like "command", "control", "option", "shift", "return", along with character keys.

scroll

Scrolls in a direction. Input: direction can be "up", "down", "left", or "right" and clicks specifies the number of scroll steps.

getScreenSize

Returns the main screen width and height.

getPixelColor

Returns RGBA color values (0-255) for the pixel at coordinates x and y.

captureScreen

Captures the entire screen and returns a base64-encoded JPEG image. Optional inputs: quality (0.0-1.0, default 0.1) and scale (0.1-1.0, default 0.25).

captureRegion

Captures a specified region defined by x, y, width, and height and returns a base64-encoded JPEG image. Optional inputs: quality and scale with the same ranges as captureScreen.

saveScreenshot

Captures the screen or a specified region and saves it to a file. Inputs include filename, optional x, y, width, height, and optional quality (JPEG only) and scale.

executeAppleScript

Executes AppleScript code. Input: script string. Returns a formatted result or a success message for no return.

executeAppleScriptFile

Executes AppleScript from a file path and returns a result or a success message with the path.