Atla

An MCP server implementation providing a standardized interface for LLMs to interact with the Atla API.

python

17

GitHub Stars

python

Language

6 months ago

First Indexed

2 months ago

Catalog Refreshed

Documentation & install

Readme and setup notes from the catalogue, plus a client-ready config you can copy for your MCP host.

Installation

Add the following to your MCP client configuration file.

Configuration

View docs

{
  "mcpServers": {
    "atla-ai-atla-mcp-server": {
      "command": "uvx",
      "args": [
        "atla-mcp-server"
      ],
      "env": {
        "ATLA_API_KEY": "YOUR_API_KEY"
      }
    }
  }
}

You have an MCP server that exposes Atla’s evaluation tools through a standardized interface so you can evaluate LLM outputs using the Atla evaluation models. It runs locally via a simple command line workflow and connects to clients that support MCP servers, enabling practical evaluation workflows for model responses and critiques.

How to use

To use the Atla MCP server, start the server in a local environment and connect your MCP client to it. The server exposes evaluation tools you can invoke from your client to score and critique an LLM’s response.

How to install

Prerequisites: ensure you have a Python-friendly environment and a way to run MCP servers locally. You will manage the runtime with a utility called uv, which helps you run MCP servers from your terminal.

Install the runtime tool for MCP servers if you do not have it yet.
Obtain your Atla API key from your Atla account.
Start the Atla MCP server using the runtime with your API key available in the environment.

ATLA_API_KEY=<your-api-key> uvx atla-mcp-server

# Example when you already have ABIs in place and uvx installed, this runs the server locally.

Additional notes

This MCP server is designed to interact with Atla’s evaluation models. Please note that the Atla API is no longer active as of July 21, 2025, so certain live evaluation features may be unavailable or require alternative access arrangements.

The server provides two core tools you can call from your MCP client to evaluate LLM responses: evaluate_llm_response and evaluate_llm_response_on_multiple_criteria.

Server connection details

The server runs as a local stdio MCP server. You start it via the runtime command, and you connect your MCP client to this local server.

Command to start the server:

ATLA_API_KEY=<your-api-key> uvx atla-mcp-server


Connection is established by your MCP client by opening a stdio channel to the running server.

Available tools

evaluate_llm_response

Evaluate an LLM's response to a prompt using a given evaluation criteria. Returns a dictionary with a score and a textual critique.

evaluate_llm_response_on_multiple_criteria

Evaluate an LLM's response across multiple evaluation criteria. Returns a list of dictionaries with scores and critiques per criterion.