FreeCrawl

A production-ready mcp server for web scraping and document processing. Drop-in replacement for FireCrawl.

python

GitHub Stars

python

Language

6 months ago

First Indexed

2 months ago

Catalog Refreshed

Documentation & install

Readme and setup notes from the catalogue, plus a client-ready config you can copy for your MCP host.

Installation

Add the following to your MCP client configuration file.

Configuration

View docs

{
  "mcpServers": {
    "dylan-gluck-freecrawl-mcp": {
      "command": "uvx",
      "args": [
        "freecrawl-mcp"
      ],
      "env": {
        "FREECRAWL_CACHE": "true",
        "FREECRAWL_API_KEYS": "key1,key2,key3",
        "FREECRAWL_HEADLESS": "true",
        "FREECRAWL_CACHE_DIR": "/tmp/freecrawl_cache",
        "FREECRAWL_CACHE_TTL": "3600",
        "FREECRAWL_LOG_LEVEL": "INFO",
        "FREECRAWL_ROTATE_UA": "true",
        "FREECRAWL_TRANSPORT": "stdio",
        "FREECRAWL_CACHE_SIZE": "536870912",
        "FREECRAWL_RATE_LIMIT": "60",
        "FREECRAWL_ANTI_DETECT": "true",
        "FREECRAWL_MAX_BROWSERS": "3",
        "FREECRAWL_MAX_CONCURRENT": "10",
        "FREECRAWL_MAX_PER_DOMAIN": "3",
        "FREECRAWL_BLOCKED_DOMAINS": "localhost,127.0.0.1",
        "FREECRAWL_REQUIRE_API_KEY": "false"
      }
    }
  }
}

FreeCrawl MCP Server is a self-hosted, production-ready MCP server for web scraping and document processing. It combines JavaScript-enabled scraping, smart caching, per-domain rate limiting, and robust error handling to give you a reliable, configurable MCP backend you can run locally or in your environment.

How to use

To use FreeCrawl with your MCP client, run the server locally and connect your client to it via the MCP protocol. You can also manage the server from your local development environment or through a lightweight runtime like uvx. The server exposes a set of MCP tools you can invoke to scrape pages, batch process URLs, extract structured data, process documents, and check health.

Use the following configuration to run FreeCrawl as an MCP server through a standard runtime. This is the canonical command setup for local or lightweight hosting.

{
  "mcpServers": {
    "freecrawl": {
      "command": "uvx",
      "args": ["freecrawl-mcp"]
    }
  }
}

Available tools

freecrawl_scrape

Scrape content from a single URL with options for formats, JavaScript, anti-detection, and caching. Returns formats like markdown, html, text, screenshot, and structured data.

freecrawl_batch_scrape

Scrape multiple URLs concurrently with configurable concurrency, formats, and error handling. Supports continuing on error and applying common options to all URLs.

freecrawl_extract

Extract structured data from a URL using a schema. Supports single or multiple item extraction, optional validation, and prompt customization.

freecrawl_process_document

Process documents (PDF, DOCX, etc.) with optional OCR and table extraction. Supports multiple formats and language options.

freecrawl_health_check

Check server health status and metrics to monitor performance and reliability.

Built by

VeilStrat

AI signals for GTM teams