Spider

Spider MCP Server is a dedicated service that crawls documentation websites, extracts clean text, and uses large language models to analyze content and generate intelligent summaries and context through the Model Context Protocol. It helps you build searchable, well-structured documentation insights from sprawling sites with configurable depth, smart content extraction, and robust caching and reliability features.

How to use

You run the Spider MCP Server locally and connect an MCP-compatible client to access its tools. This enables you to crawl target documentation sites, cache extracted content, and perform LLМ-powered analysis to obtain summaries, key points, and code examples. Use it to build a smart documentation assistant workflow where your client issues tool requests like crawling a site, retrieving cached pages, searching content, or triggering specialized analyses.

How to install

Prerequisites: ensure you have a JavaScript runtime environment and package manager available. You will use Bun to install dependencies and run the server. If you do not have Bun installed, install it from its official source before proceeding.

Step by step commands you will run locally:

# Clone the project repository
git clone <repository-url>

# Navigate to the project directory
cd spider-mcp

# Install dependencies with Bun
bun install

# Prepare environment configuration (copy example, then fill in values)
cp .env.example .env

Additional content and notes

Configuration and operation guide: you can customize how aggressively the server crawls sites, how content is extracted, and how aggressively you apply LLM analysis. The server supports two main modes: local, via stdio, and remote endpoints, via HTTP. For local usage, you typically run the dev command during development and connect clients through the stdio transport.

Configuration and running tips

To start exposing MCP tools from a local runtime, you typically invoke the server in a development-friendly way. The recommended approach shown in examples is to use the Bun runner to launch the server so that it can communicate over stdio with an MCP client.

Security and access control

Manage access by keeping your Anthropic API key secure and only exposing the MCP interface to trusted clients. When using LLM-powered features, store API keys in environment variables and avoid embedding sensitive credentials in client configurations.

Troubleshooting

If you encounter connection issues, verify your MCP client is configured to use the correct transport (stdio for local runs). Check that your environment variable ANTHROPIC_API_KEY is set if you intend to use LLM features. Review crawl settings such as max depth and robots.txt handling to ensure the target site is reachable under your policy.

Available tools

spider_docs

Crawl a documentation site, cache content, and optionally run LLМ analysis to extract summaries, key points, and code examples.

get_page

Retrieve a specific cached page by URL.

search_docs

Search through cached documentation content with optional result limits.

list_pages

List all cached pages with optional filtering and sorting.

clear_cache

Clear cached pages based on a URL pattern.

analyze_content

Perform LLM-powered analysis on a cached page with selectable analysis types.

get_summary

Get an intelligent summary of a cached page with adjustable length and focus areas.

code_examples

Extract and categorize code examples from documentation across languages.