Crawl4AI

You run an MCP server that connects to a Crawl4AI instance, letting you drive web crawling, content extraction, and browser automation from your MCP client. This guide shows practical steps to use the Crawl4AI MCP server, install it locally, and configure access from your MCP clients.

How to use

You integrate the Crawl4AI MCP server with your MCP client to perform tasks such as extracting content, taking screenshots, generating PDFs, crawling pages, and executing JavaScript. Start with a running Crawl4AI server, then add a server entry in your MCP client configuration. Use the provided environment variables to point to your Crawl4AI endpoint and to supply optional credentials.

How to install

Prerequisites you need before installing:
- Node.js 18+ and npm
- A running Crawl4AI server (either locally or on a remote host)

Step-by-step installation flow if you want to run the MCP server locally:

1. Start a Crawl4AI server (for example, using Docker):
2. Install the MCP server package for Crawl4AI and set up environment variables
3. Run development or production commands as needed

Configuration

Connection options include an HTTP-based remote server URL or a local stdio command. You can provide a base URL for the Crawl4AI server and optional authentication keys. Use these configuration blocks to connect from your MCP client.

Development

To develop and test locally with Crawl4AI, you typically run the Crawl4AI server in a container and load the MCP server during development. Use the development commands to run tests and build the MCP server, and ensure your environment variables are loaded for integration tests.

Notes and troubleshooting

If you encounter issues during integration testing, verify that the Crawl4AI server is healthy and reachable at the configured base URL. Ensure required environment variables are set, and use the provided npm scripts for integration tests to ensure proper module loading and ES module support.

Available tools

get_markdown

Extracts content from a URL as Markdown with filtering options such as raw, fit, bm25, or llm.

capture_screenshot

Captures a webpage screenshot and returns a base64-encoded PNG. Stateless; use crawl with screenshot for post-JS results.

generate_pdf

Converts a webpage to a PDF and returns a base64-encoded PDF. Stateless; use crawl for post-JS results.

execute_js

Executes JavaScript on a loaded page and returns results. Stateless; use crawl for persistent JS execution.

batch_crawl

Crawls multiple URLs concurrently with configurable parallelism and per-URL configurations.

smart_crawl

Auto-detects content type and processes HTML, sitemap, or RSS accordingly, with smart defaults.

get_html

Returns sanitized HTML optimized for structural analysis and schema-building tasks.

extract_links

Extracts and categorizes links from a page into internal, external, documents, images, and social links.

crawl_recursive

Deep crawl a website by following internal links up to a specified depth.

parse_sitemap

Extract URLs from XML sitemaps with optional regex filtering.

crawl

Advanced web crawling with an extensive set of browser, network, and extraction options.

manage_session

Unified tool for creating, listing, and clearing persistent browser sessions.

extract_with_llm

AI-driven extraction of structured data from webpages using an LLM.