Scorable

You can run a Scorable MCP Server to expose Scorable evaluators as MCP tools for AI assistants and agents. It connects your AI workflows to Scorable evaluations, letting you list evaluators, run evaluations, and measure how well responses adhere to coding and reasoning policies using a standardized MCP interface.

How to use

To use the Scorable MCP Server, first obtain your API key from your Scorable account. Then run the server locally or deploy it to your hosting environment. Once the server is running, connect your MCP client (such as Cursor) using the SSE transport endpoint. You can list evaluators, run evaluations by ID or name, run coding policy adherence checks, and work with judges that group evaluators for larger evaluation schemes.

Typical client interactions include discovering available evaluators, executing evaluations with your current prompt and model response, and reading back the scores or feedback to improve future responses. The server exposes a small set of clear tools that correspond to common evaluation tasks: listing evaluators, running evaluations, and working with judges.

How to install

Prerequisites you need before starting: a capable Node.js/npm environment if you plan to run a local build, and access to the Internet to fetch dependencies. You also need a valid API key from Scorable to authorize API calls.

Option A — run via HTTP(S) endpoint (recommended for deployment): you will run the MCP server as a remote service and connect clients over SSE.

Option B — run via stdio (local server): you execute a local MCP server process that accepts commands from your MCP host.

Additional sections

Configuration and usage notes: the server exposes the following tools for use by your MCP clients: list_evaluators, run_evaluation, run_evaluation_by_name, run_coding_policy_adherence, list_judges, and run_judge. Use these tools to retrieve evaluators, run standard evaluations, run policy-based evaluations, and manage judges.

Security tip: keep your API key secure and avoid exposing it in client-side code or public repositories. Rotate keys if you suspect exposure. Use environment variables to inject keys at runtime where possible.

Notes on endpoints: the server supports an SSE endpoint for client connections and exposes a ready-to-use path for MCP clients. The exact endpoint to connect is /sse when using the plain SSE transport. In some deployments you may also see a shorthand or alternative path like /mcp depending on your environment. Always ensure your client points to the correct transport endpoint that your deployment exposes.

Tooling and endpoints

The server provides the following tools to evaluate and manage responses:

• list_evaluators — Lists all available evaluators on your Scorable account
• run_evaluation — Runs a standard evaluation using a specified evaluator ID
• run_evaluation_by_name — Runs a standard evaluation using a specified evaluator name
• run_coding_policy_adherence — Runs a coding policy adherence evaluation using policy documents such as AI rules files
• list_judges — Lists all available judges on your Scorable account
• run_judge — Runs a judge using a specified judge ID

Security and operations

Protect your API key, use TLS in transit, and monitor server logs for unusual activity. If you deploy in a multi-tenant environment, ensure access controls are in place so only authorized clients can reach the MCP endpoint.

Available tools

list_evaluators

Lists all available evaluators on your Scorable account.

run_evaluation

Runs a standard evaluation using a specified evaluator ID.

run_evaluation_by_name

Runs a standard evaluation using a specified evaluator name.

run_coding_policy_adherence

Runs a coding policy adherence evaluation using policy documents such as AI rules files.

list_judges

Lists all available judges on your Scorable account. A judge is a collection of evaluators forming LLM-as-a-judge.

run_judge

Runs a judge using a specified judge ID.