Crawl

You have a modular MCP server that exposes 11 tools for web crawling, content extraction, and web search. It orchestrates browser automation, converts HTML to clean Markdown optimized for large language models, and can be controlled locally or over HTTP. This enables you to build end‑to‑end workflows that search the web, fetch and transform pages, and feed results into AI pipelines with robust error handling and caching.

How to use

To use the crawl MCP server, you connect a compatible MCP client and invoke the 11 built‑in tools. You can search the web using a web search tool, launch and manage a browser session for advanced interactions, and crawl or read pages to convert them into Markdown-ready content. Use the search tool to discover URLs, then feed those URLs into the crawling tools to retrieve structured Markdown suitable for AI consumption. The tools support concurrent processing, timeouts, and error handling so you can build resilient workflows.

How to install

Prerequisites you need before installation: Node.js 18 or higher, and npm or yarn for dependency management.

# Install from npm (recommended)
npm install crawl-mcp-server

# Install latest from Git
npm install git+https://github.com/Git-Fg/searchcrawl-mcp-server.git
# Or specific branch
npm install git+https://github.com/Git-Fg/searchcrawl-mcp-server.git#main
# From a fork
npm install git+https://github.com/YOUR_FORK/searchcrawl-mcp-server.git

# Clone and build
git clone https://github.com/Git-Fg/searchcrawl-mcp-server.git
cd crawl-mcp-server
npm install
npm run build

# Run without installation
npx git+https://github.com/Git-Fg/searchcrawl-mcp-server.git

Additional setup and configuration

If you want to integrate with Claude Desktop or other MCP clients, you can configure a local MCP server instance or use a remote/CLI approach. The following examples show concrete configurations you can paste into your client configuration.

{
  "mcpServers": {
    "crawl": {
      "command": "npx",
      "args": [
        "git+https://github.com/Git-Fg/searchcrawl-mcp-server.git"
      ],
      "env": {
        "NODE_ENV": "production"
      }
    },
    "crawl_node": {
      "command": "node",
      "args": [
        "/path/to/crawl-mcp-server/dist/index.js"
      ],
      "env": {}
    }
  }
}

Usage notes and tips

• The server runs as a standard MCP endpoint, exposing the 11 tools for crawling, searching, and content extraction. - You can run a local process in stdio mode for fast development, or configure a host to expose the MCP API over HTTP for remote clients. - When configuring a local runtime, ensure you point to the built distribution file as shown in the stdio example. - If you enable caching, results are SHA‑256 based and can gracefully fall back if a page cannot be fetched.

Troubleshooting

If a tool reports a network or timeout error, increase the per‑request timeout in the options and verify network access. For browser automation, ensure the Playwright dependencies install correctly and that the system can launch a Chromium instance. Check that your MCP client is connecting to the correct transport (stdio for local usage or HTTP for remote usage).

Available tools

search_searx

Search the web using SearXNG with automatic browser management.

launch_chrome_cdp

Launch system Chrome with remote debugging for advanced SearX usage.

connect_cdp

Connect to a remote CDP browser (Browserbase, etc.).

launch_local

Launch bundled Chromium for SearX search.

chrome_status

Check Chrome CDP status and health.

close

Close browser session (keeps Chrome CDP running).

shutdown_chrome_cdp

Shutdown Chrome CDP and clean up resources.

crawl_read

Quick single-page extraction to Markdown.

crawl_read_batch

Process 1-50 URLs concurrently and return Markdown results.

crawl_fetch_markdown

Controlled single-page extraction with full option control.

crawl_fetch

Multi-page crawling with intelligent link extraction.