MCP vLLM Benchmarking Tool

A very simple proof-of-concept mcp for running vllm benchmarks

python

4

GitHub Stars

python

Language

6 months ago

First Indexed

2 months ago

Catalog Refreshed

Documentation & install

Readme and setup notes from the catalogue, plus a client-ready config you can copy for your MCP host.

Installation

Add the following to your MCP client configuration file.

Configuration

View docs

{
  "mcpServers": {
    "eliovp-bv-mcp-vllm-benchmark": {
      "command": "uv",
      "args": [
        "run",
        "/Path/TO/mcp-vllm-benchmarking-tool/server.py"
      ]
    }
  }
}

You can run an MCP server that interactively benchmarks vLLM endpoints and compare performance across configurations. This makes it easy to run repeatable benchmarks, capture results, and analyze how different models or endpoints perform under your workload.

How to use

Install and run the MCP benchmarking server, then use an MCP client to initiate benchmarks. You can prompt the tool to run a vLLM benchmark against a specific endpoint, specify the model, and run multiple iterations to gather stable results. The server handles running the benchmark and presenting a comparison of results, while automatically treating the first iteration as a warmup and excluding it from the final comparison.

How to install

Prerequisites: you need a runtime capable of executing MCP stdio servers (such as uv) and access to a Python script that implements the benchmarking logic.

Clone the benchmarking tool repository.
Add the MCP server configuration to your MCP setup using the following snippet.

{
  "mcpServers": {
    "mcp_vllm": {
      "command": "uv",
      "args": [
        "run",
        "/Path/TO/mcp-vllm-benchmarking-tool/server.py"
      ]
    }
  }
}

Additional notes

Be aware that some outputs from vLLM can vary and may occasionally produce JSON that looks invalid. If this happens, re-run the benchmark to confirm consistency and track any fluctuations. The flow described here is intended to provide an interactive, repeatable benchmarking mechanism within MCP.

Available tools

benchmark

Runs a vLLM endpoint benchmark, collecting timing and accuracy-like metrics across multiple iterations to produce a comparative report.

ignore_warmup

Exclude the first iteration from the final results to remove warmup bias when comparing benchmarks.