Crawl

You can run a flexible MCP server that wraps the crawl4ai library to extract, analyze, and summarize content from diverse sources. This server supports local (stdio) and HTTP access, letting you perform web crawling, content extraction, AI-powered summaries, and YouTube transcript extraction through a unified interface.

How to use

Set up your MCP client to connect to either a local stdio server or a remote HTTP server. For stdio, you start the server locally and your client communicates directly with the process. For HTTP, you connect to the exposed HTTP endpoint over port 8000.

How to install

Prerequisites: Python 3.11 or newer is recommended for compatibility with related tooling. You will choose between two installation paths: a lightweight UVX-based setup or a production-ready Docker setup.

# UVX (Recommended - easiest):
# After system preparation, run the following to install and run Crawl MCP
uvx --from git+https://github.com/walksoda/crawl-mcp crawl-mcp

Docker (Production-Ready): clone the project, then build and run in Docker. You have two run modes: the default STDIO mode and an HTTP mode that serves on port 8000.

# Clone the repository
git clone https://github.com/walksoda/crawl-mcp
cd crawl-mcp

# Build and run with Docker Compose (STDIO mode)
docker-compose up --build

# Or build and run HTTP mode on port 8000
docker-compose --profile http up --build crawl4ai-mcp-http

# Or build manually
docker build -t crawl4ai-mcp .
docker run -it crawl4ai-mcp

Configuration and usage notes

The server can be referenced by clients in two ways. Use the stdio-config snippet when starting the server locally, or use the HTTP configuration when running in HTTP mode. Include a language setting to tailor transcripts and interfaces to your preferred language.

Troubleshooting and tips

If you encounter issues during installation or startup, verify that you have the required prerequisites installed and that you are using the correct start command for your chosen mode. For content-heavy sites, enable JavaScript processing and adjust timeouts as needed. If you require Japanese interface, set the language environment variable accordingly in the HTTP configuration.

Additional setup for Claude Desktop users

If you want to integrate with Claude Desktop, configure the MCP server as shown for the stdio or HTTP transport.

Available tools

crawl_url

Crawl a single page with JavaScript support to extract content and metadata.

deep_crawl_site

Explore a multi-page site to map structure, extract content, and identify entities.

crawl_url_with_fallback

Crawl with retry logic to handle transient failures and timeouts.

batch_crawl

Process multiple URLs in parallel for faster extraction.

intelligent_extract

Semantic content extraction guided by custom instructions.

auto_summarize

LLM-based summarization to reduce token usage while preserving key information.

extract_entities

Pattern-based extraction of entities like emails, phones, and URLs.

process_file

Convert PDFs and Office documents into markdown-friendly content.

extract_youtube_transcript

Retrieve transcripts from YouTube videos in multiple languages.

batch_extract_youtube_transcripts

Process multiple videos to extract transcripts efficiently.

search_google

Google search with genre-filtered queries and metadata.

search_and_crawl

Combine search results with content extraction for rapid analysis.

batch_search_google

Run multiple Google searches and analyze results in batch.