Spider

Spider MCP is a web search and page-scraping MCP server built on pure crawler technology using Node.js. It enables web and news searches without relying on official APIs, while providing health checks, structured logging, and anti-detection measures to help you fetch results with control and visibility.

How to use

You interact with Spider MCP through an MCP client. Use the unified web_search tool to perform both web searches and news searches. When you call the tool, specify the search type as either web or news and set any relevant time filters for news searches. The tool returns a list of results you can process in your workflow. Authentication or API keys aren’t required for the core crawler endpoints, but you should respect target websites’ terms of use and robots.txt.

Practical usage patterns include:

• Run a web search for a topic and extract the top 10 results for integration into a knowledge base.
• Run a news search with a time filter to surface recent articles related to a topic.
• Use the page content tools to fetch and convert a result’s page content for downstream analysis.

How to install

Prerequisites you should have before installing: Node.js 18 or newer, a modern package manager (npm or pnpm), and network access to install dependencies and download the Puppeteer browser.

npm install
"""
or
"""
pnpm install

Download the Puppeteer browser so the crawler can render pages as needed.

npx puppeteer browsers install chrome

Configure environment variables for the service. Copy the example and adjust values as needed.

cp .env.example .env
# Edit .env to set NODE_ENV, ports, and any other required vars

Start the service in development or production mode. The service will listen on http://localhost:3000 by default.

npm run dev

# or for production
npm start

Configuration and deployment notes

The service includes an MCP configuration snippet you can place in your chatbox or MCP client configuration to expose Spider MCP. The entry defines a local stdio runtime that runs the MCP server script.

{
  "mcpServers": {
    "spider-mcp": {
      "command": "node",
      "args": ["src/mcp/server.js"],
      "env": {
        "NODE_ENV": "production"
      },
      "description": "Spider MCP - Web search and webpage scraping tools",
      "capabilities": {
        "tools": {}
      }
    }
  }
}

Available tools

web_search

Unified search tool supporting both web and news search, returning results with optional time filters for news searches.

get_webpage_content

Fetch webpage content and convert it to a specified format for downstream processing.

get_webpage_source

Obtain the raw HTML source code of a webpage for parsing or analysis.

batch_webpage_scrape

Batch scrape multiple webpages in a single operation to improve throughput and efficiency.