ViperMCP: A Model Context Protocol for Viper Server

ViperMCP provides a streaming, MCP-native HTTP endpoint for sheet-moE visual tasks and image-centric questions, enabling you to route queries across multiple VLMs/LLMs and retrieve both textual answers and visual outputs like masks. It is designed to work with standard MCP tooling and client libraries, making it easy to integrate image-based reasoning into your apps.

How to use

You connect your MCP client to the ViperMCP server through the HTTP endpoint at your running server address, typically via the /mcp/ path. Use the provided tools to ask questions about images, request visual grounding, or generate masked outputs. The server supports streaming responses, so you can start receiving results as they are produced and refine prompts on the fly.

How to install

Prerequisites you need before starting: a Python environment and optionally Docker for a containerized setup. You also need an OpenAI API key to access the underlying models.

Option 1: Dockerized FastMCP Server (GPU‑ready) Run the following, after saving your key to a file named api.key:

# Save your key to api.key
# Then run the container with your key mounted
docker run -i --rm \
  --mount type=bind,source=/path/to/api.key,target=/run/secrets/openai_api.key,readonly \
  -e OPENAI_API_KEY_PATH=/run/secrets/openai_api.key \
  -p 8000:8000 \
  rsherby/vipermcp:latest

This starts a CUDA‑enabled container serving MCP at http://0.0.0.0:8000/mcp/.

Option 2: Pure FastMCP Server (dev‑friendly) Set up from source and run locally:

git clone --recurse-submodules https://github.com/ryansherby/ViperMCP.git
cd ViperMCP
bash download-models.sh

# Store your OpenAI key for local development
echo YOUR_OPENAI_API_KEY > api.key

# Recommended: use a virtual environment
python -m venv venv
source venv/bin/activate

pip install -r requirements.txt
pip install -e .

# Start the server
python run_server.py

Endpoints and how to connect

The server exposes an HTTP MCP endpoint and a local runtime you can use for development or automation.

Security and production notes

OpenAI API keys are required and must be supplied securely, via environment variables or a mounted secret file. Plan to isolate code generation and execution, and implement strict access control and secret management when deploying to production.

Available tools

viper_query

Returns a textual answer to your query based on an image and a text prompt.

viper_task

Returns a set of image outputs (such as masks or crops) that satisfy the given task on the input image.