Crawl4AI

You run this MCP server to crawl the web, extract content, and analyze results with AI-powered tools. It exposes a standardized MCP interface so you can control heavy crawling, data extraction, and reporting from your tooling or clients.

How to use

Start by choosing how you want to connect to the MCP server. You can use STDIO for a local process or HTTP for remote access. In STDIO, you run the server as a Python module and you interact with it through standard input/output. In HTTP, you start the server as a background service and call tools over HTTP to crawl, extract, and process data. Once running, you can invoke any of the available tools to perform crawling, extraction, and analysis tasks. The server supports JavaScript-heavy sites, YouTube transcript extraction, file processing to Markdown, and AI-assisted content analysis. You can also enable YouTube transcripts, batch operations, and Google Search enrichment to enhance your workflows.

Start the MCP server (two transport options)

# STDIO transport (default, local process)
python -m crawl4ai_mcp.server

# HTTP transport (remote access)
python -m crawl4ai_mcp.server --transport http --host 127.0.0.1 --port 8000

What you can do once the server is running

You will register and call MCP tools through your client or integration layer. Typical tasks include performing a site crawl, extracting entities, processing files, retrieving YouTube transcripts, and performing Google searches with metadata extraction. The server supports multiple extraction strategies, content filtering options, and structured data export. You can also enable authentication and configure headers for secure access.

Available tools

crawl_url

Advanced web crawling with deep crawling support and intelligent filtering.

deep_crawl_site

Dedicated tool for comprehensive site mapping and recursive crawling.

intelligent_extract

AI-powered content extraction with advanced filtering and analysis.

extract_entities

High-speed entity extraction using regex patterns.

extract_structured_data

Structured data extraction using CSS/XPath selectors or LLM schemas.

batch_crawl

Parallel processing of multiple URLs with unified reporting.

crawl_url_with_fallback

Robust crawling with multiple fallback strategies for maximum reliability.

process_file

File processing to convert formats to Markdown, including PDFs and Office documents.

get_supported_file_formats

Retrieve a comprehensive list of supported file formats and capabilities.

extract_youtube_transcript

Extract transcripts from YouTube videos with language options and translation.

batch_extract_youtube_transcripts

Batch processing of multiple YouTube transcripts with controlled concurrency.

get_youtube_video_info

Retrieve available transcript information for YouTube videos.

search_google

Google search with genre filtering and metadata extraction.

batch_search_google

Batch Google search with comprehensive analysis and statistics.

search_and_crawl

Integrated search plus crawl to fetch and analyze top results.

get_search_genres

List available search genres with descriptions.