Patronus

You deploy and run the Patronus MCP Server to manage powerful LLM system optimizations, evaluations, and experiments through a standardized MCP interface. This server lets you initialize projects, run single or batch evaluations with configurable evaluators, and conduct experiments on datasets, all from a programmable API.

How to use

You interact with the Patronus MCP Server by running it locally and sending requests via the MCP client tooling. Start the server with your API key or by setting an environment variable, then call the server endpoints to initialize projects, run evaluations, or execute experiments. You can run single evaluations, batch evaluations, and experiments with datasets, and you can retrieve information about available evaluators and criteria.

How to install

Prerequisites: ensure you have Python and a shell environment available on your machine.

1. Clone the project directory.

git clone https://github.com/yourusername/patronus-mcp-server.git
cd patronus-mcp-server

2. Create and activate a virtual environment.

python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

3. Install main and development dependencies.

uv pip install -e .
uv pip install -e ".[dev]"

4. Run the server with an API key either as a command-line argument or via an environment variable.

python src/patronus_mcp/server.py --api-key your_api_key_here

export PATRONUS_API_KEY=your_api_key_here
python src/patronus_mcp/server.py

Additional setup notes

If you prefer interactive testing, you can use the test suite to exercise endpoints and view formatted results. The tests demonstrate single evaluation, batch evaluation, and live interaction patterns. You can adapt these calls to your own evaluators and datasets.

Configuration and usage notes

The server accepts an API key either through a command-line option or via an environment variable. When started with a key on the command line, the server uses that key for authentication. If you choose to provide the key via an environment variable, set PATRONUS_API_KEY before launching the server.

Security considerations

Protect your API key and limit access to the server only to trusted clients. Use environment-based configuration in production and rotate keys regularly. Review the evaluators and criteria you enable to ensure they enforce the intended pass/fail semantics without exposing sensitive data.

Troubleshooting tips

If the server fails to start, verify that the virtual environment is activated, the dependencies installed correctly, and the API key is either passed on the command line or set in the environment. Check firewall settings to allow local API access if you are running clients on separate processes.

Notes and examples

You can initialize a project with a name and API key, then perform single or batch evaluations using remote evaluators. You can also run experiments that combine remote and custom evaluators with a given dataset. See the programmatic usage examples for how to structure requests and interpret responses.

Available tools

initialize

Initialize Patronus with project_name and api_key to set up a new MCP session.

evaluate

Run a single evaluation using a remote evaluator configuration with input task and context.

batch_evaluate

Run multiple evaluations in a batch, each with its own evaluator configuration.

run_experiment

Execute an experiment that combines datasets with evaluators (remote and/or custom) to produce results.

list_evaluator_info

Retrieve information about available evaluators and their criteria.

create_criteria

Create a new evaluator criteria with a specified family and configuration.

custom_evaluate

Evaluate a task output using a user-defined evaluator function decorated for MCP usage.