Scorable

You can deploy the Scorable MCP Server to evaluate AI outputs against quality criteria using Scorable evaluators. This server exposes evaluators as MCP tools and supports both HTTP/SSE deployment and local stdio usage, enabling easy integration with MCP clients like Cursor and other hosts.

How to use

You will connect your MCP client to the Scorable MCP Server to list evaluators, run evaluations, and use judges. The server exposes tools that let you discover available evaluators, execute evaluations by ID or name, and run policy or faithfulness checks. You can also run judges that group multiple evaluators.

How to install

Prerequisites: install Docker on your host if you plan to run the server in a container. If you prefer a local stdio setup, you will need a compatible runtime to execute the MCP client wrapper (for example, uvx).

# Prerequisite: Docker
# Run the Scorable MCP Server with SSE transport (HTTP endpoint)
docker run -e SCORABLE_API_KEY=<your_key> -p 0.0.0.0:9090:9090 --name=rs-mcp -d ghcr.io/scorable/scorable-mcp:latest

# Logs show the server starting and the SSE endpoint
# You can access the SSE endpoint at http://0.0.0.0:9090/sse

Additional setup notes

After you start the server, you will typically configure your MCP client to connect via the SSE endpoint or the legacy /mcp endpoint. The preferred path is the SSE endpoint at /sse when using HTTP transport. If you are using a downstream client that supports stdio, you can also run the local server via an MVCP wrapper command as described in the stdio setup.

Configuration and access

Two connection methods are shown. Use the HTTP/SSE method first for remote deployments, and fall back to the local stdio method if you want to run the server inside your own process space.

Security considerations

Protect your API key and limit access to the MCP server to trusted clients. Use environment variables to pass keys where supported, and prefer transport with HTTPS or a controlled network interface in production.

Notes on transport endpoints

The server exposes an SSE endpoint at /sse for MCP clients that support HTTP transport. The logs will indicate the host and port, and you can verify the endpoint in your client configuration.

Usage with Cursor or MCP clients

In your MCP client configuration, point to the SSE endpoint. For example, in client configuration you specify the server URL for SSE transport as the MCP endpoint and keep your API key secure.

Tools and evaluation flow

You can list available evaluators, run evaluations by ID or by name, and perform specialized checks such as coding policy adherence or RAG-based evaluation using the provided evaluator collections. You can also run judges that combine multiple evaluators to form a higher-level quality assessment.

Examples of common usage patterns

Discover evaluators, then run an evaluation with a sample pair of request and response. You can also specify contexts to provide additional information to the evaluation engine.

Troubleshooting

If the server fails to start, check that the docker container has network access, the API key is valid, and the port is not in use. Review logs for messages about fetching evaluators from the Scorable API and verify that the SSE endpoint is reachable from your MCP client.

Examples of connecting from an MCP client

{
  "mcpServers": {
    "scorable": {
      "url": "http://localhost:9090/sse"
    }
  }
}

Available tools

list_evaluators

Lists all available evaluators on your Scorable account.

run_evaluation

Runs a standard evaluation using a specified evaluator ID.

run_evaluation_by_name

Runs a standard evaluation using a specified evaluator name.

run_coding_policy_adherence

Runs a coding policy adherence evaluation using policy documents.

list_judges

Lists all available judges on your Scorable account.

run_judge

Runs a judge using a specified judge ID.