Crawl4AI

Provides web scraping and website crawling capabilities for LLM agents via MCP endpoints.

python

26

GitHub Stars

python

Language

4 months ago

First Indexed

3 weeks ago

Catalog Refreshed

Documentation & install

Readme and setup notes from the catalogue, plus a client-ready config you can copy for your MCP host.

Installation

Add the following to your MCP client configuration file.

Configuration

View docs

{
  "mcpServers": {
    "ritvij14-crawl4ai-mcp": {
      "command": "uv",
      "args": [
        "--directory",
        "/ABSOLUTE/PATH/TO/PARENT/FOLDER/crawl4ai-mcp",
        "run",
        "main.py"
      ],
      "env": {
        "UV_DIRECTORY": "\"/absolute/path/to/crawl4ai-mcp\""
      }
    }
  }
}

You can run the Crawl4AI MCP Server to enable a small set of web-scraping and crawling capabilities for use with large language models in MCP-enabled environments. This server lets you scrape a single page or crawl a website, exposing those results as tools you can call from your MCP client.

How to use

After you have the server running, you can call its tools from an MCP client in agent mode. The server exposes two main tools you can use in your prompts:

scrape_webpage(url) grabs the content and metadata from a single page and returns a structured TextContent object (primarily markdown) with the page data.
crawl_website(url, crawl_depth=1, max_pages=5) begins from a starting URL and crawls related pages up to the specified depth and page limit, returning a JSON array of results that includes each page URL, success status, and the scraped content or errors.

How to install

Prerequisites you need before starting:

Python 3.10 or higher
A Unix-like shell on macOS/Linux or PowerShell on Windows

Follow these steps to install and run the MCP server locally:

Install the uv tooling to manage environments and runtimes

curl -LsSf https://astral.sh/uv/install.sh | sh

Windows users install uv with PowerShell

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Prepare the project directory and dependencies

# Navigate to the crawl4ai-mcp directory
cd crawl4ai-mcp

# Install dependencies (Only first time)
uv venv
uv sync

# Activate the virtual environment
source .venv/bin/activate

# Run the server
python main.py

Add the MCP server configuration

To connect your MCP client to this local server, use the following configuration snippet. It runs the server through uv to ensure the correct environment is used.

{
  "mcpServers": {
    "crawl4ai": {
      "command": "uv",
      "args": [
        "--directory",
        "/ABSOLUTE/PATH/TO/PARENT/FOLDER/crawl4ai-mcp",
        "run",
        "main.py"
      ]
    }
  }
}

Available tools

scrape_webpage

Scrapes the content and metadata from a single webpage using Crawl4AI and returns a TextContent object containing the scraped content in JSON.

crawl_website

Crawls a website starting from a URL up to a specified depth and page limit, returning a JSON array of results including URL, status, and content.