Crawl4AI

customScraperMCP

python

0

GitHub Stars

python

Language

7 months ago

First Indexed

3 months ago

Catalog Refreshed

Documentation & install

Readme and setup notes from the catalogue, plus a client-ready config you can copy for your MCP host.

Installation

Add the following to your MCP client configuration file.

Configuration

View docs

{
  "mcpServers": {
    "joedank-mcpcrawl4ai": {
      "command": "npx",
      "args": [
        "-y",
        "iterm-mcp"
      ],
      "env": {
        "CRAWL4AI_BASE_URL": "http://localhost:11235"
      }
    }
  }
}

You can use this MCP server to interact with the Crawl4AI Docker API through a reliable stdio transport, enabling you to scrape, crawl, extract content, and run browser JavaScript with Claude Code.

How to use

Connect your MCP client to the Crawl4AI MCP Server to perform a range of web automation tasks. You can crawl multiple URLs, extract clean markdown, grab HTML structures, take full-page screenshots, generate PDFs, execute JavaScript in a browser context, and query Crawl4AI documentation and examples. Once connected, send prompts to invoke any of the available tools and receive structured results.

How to install

{
  "mcpServers": {
    "iterm-mcp": {
      "args": ["-y", "iterm-mcp"],
      "command": "npx"
    },
    "crawl4ai-custom": {
      "command": "/Volumes/4TB/Users/josephmcmyne/myProjects/mcp/crawl4ai-wrapper/venv/bin/python",
      "args": ["/Volumes/4TB/Users/josephmcmyne/myProjects/mcp/crawl4ai-wrapper/crawl4ai_mcp.py"],
      "env": {
        "CRAWL4AI_BASE_URL": "http://localhost:11235"
      }
    }
  }
}

Additional setup and Notes

Prerequisites you need before starting: Python 3.8 or higher, Docker, and Claude Code installed.

Start Crawl4AI Docker container on port 11235.

docker run -d -p 11235:11235 --name crawl4ai --shm-size=2g unclecode/crawl4ai:latest

Available tools

scrape_markdown

Extract clean markdown from a webpage using crawl4ai capabilities.

extract_html

Get preprocessed HTML structures from a webpage.

capture_screenshot

Capture a full-page PNG screenshot of a URL.

generate_pdf

Create a PDF document from a webpage.

execute_javascript

Run JavaScript code within the context of a webpage.

crawl_urls

Crawl multiple URLs with configurable browser and crawler settings.

ask_crawl4ai

Query Crawl4AI documentation and examples to retrieve information.