VisionAgent

VisionAgent MCP Server lets you run a local, stdio-based MCP server that translates your MCP client’s tool calls into authenticated HTTPS requests to Landing AI’s VisionAgent REST APIs. It streams back the response JSON and any images or masks, so you can issue natural-language computer-vision and document-analysis commands directly from your editor or client without writing REST code or loading an SDK.

How to use

You use VisionAgent MCP Server by starting it as a local MCP service and configuring your MCP client to talk to it. The server accepts tool calls from compatible clients, validates inputs, forwards authenticated requests to VisionAgent, and returns results along with optional generated media. If you enable image display, the server saves visual outputs (like bounding boxes, masks, or depth maps) to a local directory and provides paths or previews to your client.

How to install

Prerequisites you need before installation are listed here so you can set up a running environment.

# 1  Install the MCP client tooling (VisionAgent MCP)
npm install -g vision-tools-mcp

# 2  Prepare your MCP client configuration (example shown within the MCP config snippet)
# The following is a complete config block you will place in your MCP client setup

CD to configuration and run setup

Configure your MCP client to run VisionAgent MCP by defining the server under mcpServers with the stdio transport. Use the example below to substitute your actual API key and paths.

{
  "mcpServers": {
    "VisionAgent": {
      "command": "npx",
      "args": ["vision-tools-mcp"],
      "env": {
        "VISION_AGENT_API_KEY": "<YOUR_API_KEY>",
        "OUTPUT_DIRECTORY": "/path/to/output/directory",
        "IMAGE_DISPLAY_ENABLED": "true"
      }
    }
  }
}

Run the MCP client and start using it

Open your MCP-aware client and connect to the VisionAgent MCP server you configured. Once connected, you can issue prompts that map to tool capabilities like document analysis, object detection, segmentation, activity recognition in video, or depth estimation.

Additional sections

The VisionAgent MCP Server forwards calls to VisionAgent endpoints using an authenticated HTTP request. It validates input against schemas derived from the live OpenAPI spec, handles file-based arguments by encoding them to base64 when needed, and returns a structured MCP response. If image rendering is enabled, outputs such as masks, bounding boxes, and depth maps are saved to the OUTPUT_DIRECTORY and paths to these artifacts are provided in the response.

Configuration

Environment variables you’ll typically configure are shown below. They are required or optional as indicated.

{
  "mcpServers": {
    "VisionAgent": {
      "command": "npx",
      "args": ["vision-tools-mcp"],
      "env": {
        "VISION_AGENT_API_KEY": "<YOUR_API_KEY>",
        "OUTPUT_DIRECTORY": "/path/to/output/directory",
        "IMAGE_DISPLAY_ENABLED": "true"
      }
    }
  }
}

Security & privacy

The MCP server runs locally on your machine, and outputs are written to your local OUTPUT_DIRECTORY. No telemetry is collected by this project, and you only interact with Landing AI VisionAgent endpoints you explicitly call.

Troubleshooting

If authentication fails, verify your API key is correct and active. If you see a 'tool not found' message, regenerate the tool map with your VisionAgent updates. Ensure your environment allows outbound HTTPS to the VisionAgent API host.

Notes on usage patterns

Run new endpoints by regenerating the local tool map when VisionAgent releases new endpoints. Use the command npm run generate-tools to refresh definitions, then restart the MCP server to pick up changes.

Available tools

agentic-document-analysis

Parse PDFs/images to extract text, tables, charts, and diagrams while preserving layouts and visual cues.

text-to-object-detection

Detect objects from free-form prompts and output bounding boxes using supported models.

text-to-instance-segmentation

Generate pixel-perfect masks for objects in images using segmentation models.

activity-recognition

Recognize activities in video with start/end timestamps.

depth-pro

Produce high-resolution monocular depth maps for single images.