Web Crawl

Provides web crawl search, filtering, and analysis across multiple crawlers for efficient data exploration.

python

34

GitHub Stars

python

Language

6 months ago

First Indexed

2 months ago

Catalog Refreshed

Documentation & install

Readme and setup notes from the catalogue, plus a client-ready config you can copy for your MCP host.

Installation

Add the following to your MCP client configuration file.

Configuration

View docs

{
  "mcpServers": {
    "pragmar-mcp-server-webcrawl": {
      "command": "mcp-server-webcrawl",
      "args": [
        "--crawler",
        "wget",
        "--datasrc",
        "/path/to/datasrc",
        "--interactive"
      ]
    }
  }
}

You can search, filter, and analyze web crawl data with mcp-server-webcrawl. It provides a powerful command-line interface and a flexible query system that lets you explore crawled content, run lightweight analytics prompts, and assemble your own knowledge base from web sources. This server is designed to work with Claude Desktop and Python-based tooling to give you precise, token-efficient results for web archives and crawled assets.

How to use

Start the server from your terminal and connect it to your preferred MCP client workflow. You can run it in an interactive terminal session to perform Boolean fulltext searches across crawled data, or use prompts to execute analytical routines like SEO audits, 404 audits, performance checks, and file analyses. The system supports multiple crawlers and allows you to filter results by type, HTTP status, and other fields. When you have a query, you can refine results with field searches (id, url, type, status, headers, content) and extras like markdown or snippets to optimize token usage.

How to use the CLI in practice

Install the package with Python’s pip tool: pip install mcp-server-webcrawl
Launch an interactive search session against a local or remote crawl by running a single command, then enter queries directly in the console
Example: start a search session against a data source using a specific crawler and dataset
Use prompts to run predefined analyses such as SEO Audit, 404 Audit, or Performance Audit to get structured insights
Switch between crawled sites or data sources by changing the active datasrc and crawler in the UI or via command flags

Available tools

SEO Audit

Technical SEO analysis with options to dive deeper, focusing on search optimization for crawled content.

404 Audit

Broken link detection and pattern analysis with actionable fixes.

Performance Audit

Website speed and optimization analysis with practical recommendations.

File Audit

File organization and asset analysis to reveal the structure of your site.

Gopher Interface

A classic text-based search interface for querying crawled data.

Search Test

A self-check battery to validate boolean query parsing and full-text conversion.