mcp-com-ai/mcp-server-evaluations-skills
Overview
This skill tests MCP servers for quality and reliability by running health checks, discovering exposed tools, exercising functionality, and scoring response quality. It requires only curl and focuses on verifying tool completeness, error handling, and question-based accuracy. Use it to validate MCP implementations or OpenAPI-to-MCP conversions before production use.
How this skill works
The skill performs a phased evaluation: environment verification (health and ping), tool discovery (list and inventory), functional tests (valid and invalid calls), question-based evaluations, and a final quality score. It issues JSON-RPC requests against the /mcp endpoint using curl, inspects responses for schema correctness, and assesses error messages and performance. Results are collated into a weighted score across discovery, functionality, error handling, accuracy, and performance.
When to use it
- Validating a new MCP server deployment before production rollout
- Testing OpenAPI-to-MCP conversions to ensure all operations were exposed
- Auditing API tool quality and consistency across teams
- Troubleshooting unexpected tool behavior or error responses
- Benchmarking response times and error clarity for SLAs
Best practices
- Start with the quick evaluation checklist to catch obvious failures fast
- Document discovered tools and expected parameter schemas before testing
- Run both valid and invalid argument calls to verify graceful error handling
- Generate realistic, multi-step questions to exercise tool orchestration
- Measure response times consistently and repeat tests for variability
Example use cases
- Smoke test a staging MCP server: health, ping, list tools, sample call
- Confirm every OpenAPI operation appears as a tool with correct parameter types
- Assess error message quality by calling tools with missing required fields
- Run a suite of 10+ user-style questions to score correctness and completeness
- Calculate an overall pass/fail using the weighted scoring rubric (80% pass)
FAQ
No. The checks and test templates are designed to run with curl only; optional tools like jq can make output parsing easier.
What does a failing error-handling test look like?
Failures include unhelpful or generic error messages, missing identification of required parameters, incorrect HTTP status propagation, or crashes on invalid input.
