Mandoline

Mandoline MCP Server lets you integrate evaluation tools into your AI assistants so they can compare, critique, and improve their own responses over time using the Model Context Protocol. You can use a hosted MCP server for quick setup or run the server locally for development and contribution work.

How to use

Connect your client to the Mandoline MCP Server to access evaluation tools during conversations. Use a hosted MCP URL to quickly start evaluating prompt/response quality with predefined metrics and evaluations. When you configure a client, provide your API key so Mandoline can authorize requests. After setup, you can invoke evaluation and health tools directly during interactions to score prompts, create metrics, and retrieve evaluation results.

How to install

Prerequisites: you need Node.js version 18 or newer and npm installed on your system.

# 1) Install dependencies and build (local server)
git clone https://github.com/mandoline-ai/mandoline-mcp-server.git
cd mandoline-mcp-server
npm install
npm run build

# 2) Prepare environment (optional)
cp .env.example .env.local
# Edit .env.local to customize PORT, LOG_LEVEL, and other settings
```} ,{

Start the local server when you are ready to run and test MCP locally.

npm start

Configuration and notes

Client connections can use either the hosted server or a local MCP server. The hosted server URL is https://mandoline.ai/mcp and requires an API key to authorize access. Local usage runs on your machine and serves requests from http://localhost:8080/mcp by default. If you run locally, you can customize the port and logging level via environment settings.

Environment variables you may encounter when configuring the server or clients include:

PORT=8080
LOG_LEVEL=info
# Optional: API key placeholder for local tests
MANDOLINE_API_KEY=sk_****

Tools and usage notes

Mandoline exposes tools to health-check the server, create and manage evaluation metrics, and run evaluations. Common tools include:

• get_server_health: checks that the MCP server is reachable and returns a healthy status payload.
• create_metric, batch_create_metrics, get_metric, get_metrics, update_metric: manage evaluation metrics.
• create_evaluation, batch_create_evaluations, get_evaluation, get_evaluations, update_evaluation: perform evaluations and review results.
• Resources provide docs indexes and MCP setup guidance.

Available tools

get_server_health

Check MCP server reachability and health status.

create_metric

Define a new evaluation metric to quantify task performance.

batch_create_metrics

Create multiple metrics in a single operation.

get_metric

Retrieve details about a specific metric.

get_metrics

Browse metrics with filtering and pagination.

update_metric

Modify an existing metric definition.

create_evaluation

Score prompt/response pairs against your metrics.

batch_create_evaluations

Evaluate the same content against multiple metrics in one call.

get_evaluation

Retrieve results and scores for a single evaluation.

get_evaluations

Browse evaluation history with filtering and pagination.

update_evaluation

Add metadata or context to evaluations.