MCP as a Judge

Provides an MCP‑based safety and quality gate that evaluates plans, code changes, and tests with evidence-based judgments.

typescript

GitHub Stars

typescript

Language

4 months ago

First Indexed

2 months ago

Catalog Refreshed

Documentation & install

Readme and setup notes from the catalogue, plus a client-ready config you can copy for your MCP host.

Installation

Add the following to your MCP client configuration file.

Configuration

View docs

{
  "mcpServers": {
    "othervibes-mcp-as-a-judge": {
      "command": "docker",
      "args": [
        "run",
        "--rm",
        "-i",
        "--pull=always",
        "ghcr.io/othervibes/mcp-as-a-judge:latest"
      ],
      "env": {
        "LLM_API_KEY": "YOUR_OPENAI_API_KEY",
        "LLM_MODEL_NAME": "gpt-4o-mini"
      }
    }
  }
}

MCP as a Judge provides an evidence-driven validation layer between AI coding assistants and LLMs. It enforces plan, code changes, and tests through explicit evaluations, ensuring safer, higher-quality software development with human-in-the-loop decision making.

How to use

You use MCP as a Judge by configuring an MCP-enabled client to run the judge server and then perform plan and code evaluations, design reviews, and test validations as part of your workflow. The judge adds enforceable gates that require approvals before proceeding to the next stage, and it surfaces actionable feedback when content evaluation detects risks or gaps.

How to install

Prerequisites: install Docker Desktop and ensure Python 3.13 or newer is available on your machine. You also need an MCP-enabled client that can connect to a local or remote MCP server.

Configuration and usage notes

Two common ways to run MCP as a Judge are provided below. Choose the method that best fits your environment and client.

Security and privacy

MCP as a Judge runs locally on your machine so your code and conversations stay private. No data is collected or sent externally when using MCP Sampling. If you enable a fallback LLM API, the server will call the chosen provider only to perform judgments with the evaluation content you provide.

Troubleshooting

If the MCP server isn’t auto-used, check that your client is configured to use the server, ensure the server process is running, and verify that the judge tools are enabled in your client.

Tools and capabilities

The judge supports plan evaluation, code change review, test validation, and completion gating. It also includes elicitation and risk flagging to guide decisions when requirements are ambiguous or constraints exist.

Available tools

set_coding_task

Creates or updates task metadata, classifies task size, and returns the next-step guidance for the workflow.

get_current_coding_task

Recovers the latest task identifier and metadata to resume work safely.

judge_coding_plan

Validates the plan or design, requiring library choices and reuse maps, and flags risks.

judge_code_change

Reviews unified diffs for correctness, reuse, security, and code quality.

judge_testing_implementation

Validates tests using actual runner outputs and optional coverage information.

judge_coding_task_completion

Final gate ensuring plan, code, and tests are approved before completion.

raise_missing_requirements

Elicits missing details and decisions to unblock progress.

raise_obstacle

Engages the user on trade-offs, constraints, and enforced changes.

Built by

VeilStrat

AI signals for GTM teams