Crawlab

You can run the Crawlab MCP Server to expose Crawlab features to AI applications via a standardized, programmable interface. It lets you manage spiders and tasks, read and write files, and access resources through a consistent MCP protocol so your AI workflows can orchestrate Crawlab actions smoothly.

How to use

Start the MCP server and connect your MCP client to control Crawlab programmatically. Use natural language prompts in your AI client to perform actions like creating and managing spiders, running tasks, and reading or saving files. The server translates your requests into Crawlab API calls and returns results in a structured, human-readable format.

How to install

Prerequisites: Python 3.8 or higher and a Crawlab instance accessible via API. You should also have an API token for Crawlab.

Option 1: Install as a Python package and run the MCP server from the CLI.

# Install from source
pip install -e .

# Or install from a Git source when available
# pip install git+https://github.com/crawlab-team/crawlab-mcp-server.git

Configuration and starting options

Two practical ways to run the MCP server are provided. The first uses a dedicated CLI to start the server, and the second runs a Python script locally for development or testing.

CLI start (server mode) requires specifying optional spec, host, and port parameters when you start the server.

crawlab_mcp-mcp server [--spec PATH_TO_SPEC] [--host HOST] [--port PORT]

Local run (development)

If you prefer starting the MCP server directly from Python for development, you can run the server script. Ensure you have the necessary dependencies installed first.

python server.py

Environment variables to configure Crawlab API access

Create and edit a local environment configuration file to point the MCP server at your Crawlab instance. The following variables are used to connect to Crawlab API.

# .env.example
CRAWLAB_API_BASE_URL=http://your-crawlab-instance:8080/api
CRAWLAB_API_TOKEN=your_api_token_here

Starting with Docker Compose

If you are integrating into an existing Crawlab deployment with Docker Compose, add an MCP server service to your compose file. The service will expose the MCP API on a chosen port and pass the necessary environment variables to connect to Crawlab.

Examples of common workflows

Create a spider, run a task, and read logs using natural language prompts in your MCP-enabled AI client. The MCP layer translates these prompts into Crawlab API actions and returns results.

Available tools

get_spider

Retrieve details for a specific spider by its identifier.

create_spider

Create a new spider with given specifications and project association.

update_spider

Update properties of an existing spider, such as its code or configuration.

delete_spider

Remove a spider from Crawlab.

get_task

Fetch details for a specific task or run.

run_spider

Execute a spider across the requested scope (nodes, environments, or schedules).

cancel_task

Cancel a currently running task.

restart_task

Restart a previously executed task.

get_task_logs

Retrieve logs for a specific task to diagnose results.

get_spider_files

List all files associated with a spider.

get_spider_file

Read the contents of a specific file in a spider.

save_spider_file

Write or update the contents of a spider file.