Web Docs

You run a self-hosted MCP server that crawls, indexes, and searches documentation from any website, including private docs behind authentication. This lets you build fast, private knowledge bases you can query with ease from your MCP client of choice.

How to use

You index documentation sites, then search across them using an MCP client. Start by registering one or more documentation sites you want to crawl. You can index public docs, private docs (with interactive authentication), and multiple versions of the same package. Use a collection to group related sites for projects, and apply tags to filter searches.

Core actions you perform through the MCP client are: add a documentation site to index, search indexed content with a hybrid full-text and semantic search, authenticate to protected docs, manage collections, and reindex as docs update. For best results, craft precise queries, optionally filter by site URL or tags, and leverage versioning when indexing multi-version packages.

How to install

Prerequisites: ensure Node.js is installed (version 22.19.0 or newer). You will also need a terminal to run commands.

Option 1: Install from npm (recommended) — global install.

Install from npm (recommended)

npm install -g @cosmocoder/mcp-web-docs

Option 2: Run with npx

No installation required. Use npx to run the MCP server directly when you want to index docs.

Option 3: Build from source

# Clone the repository
git clone https://github.com/cosmocoder/mcp-web-docs.git
cd mcp-web-docs

# Install dependencies (this also installs Playwright browsers)
npm install

# Build
npm run build

Configure your MCP client

You connect your MCP client to the web-docs server by providing a command or URL that starts the server. Below are example configurations for common clients. Choose the method that matches how you want to run the server (npx, global install, or local build).

Cursor

{
  "mcpServers": {
    "web-docs": {
      "command": "npx",
      "args": ["-y", "@cosmocoder/mcp-web-docs"]
    }
  }
}

Claude Desktop

{
  "mcpServers": {
    "web-docs": {
      "command": "npx",
      "args": ["-y", "@cosmocoder/mcp-web-docs"]
    }
  }
}

VS Code

{
  "servers": {
    "web-docs": {
      "command": "npx",
      "args": ["-y", "@cosmocoder/mcp-web-docs"]
    }
  }
}

Windsurf

{
  "mcpServers": {
    "web-docs": {
      "command": "npx",
      "args": ["-y", "@cosmocoder/mcp-web-docs"]
    }
  }
}

Cline

Add to the configured path with the following example to enable the web-docs MCP server.

{
  "mcpServers": {
    "web-docs": {
      "command": "npx",
      "args": ["-y", "@cosmocoder/mcp-web-docs"]
    }
  }
}

RooCode

Global configuration for RooCode can be edited to include the web-docs MCP server as a running process.

{
  "mcpServers": {
    "web-docs": {
      "command": "npx",
      "args": ["-y", "@cosmocoder/mcp-web-docs"]
    }
  }
}

Quick start

Index the LanceDB documentation, then search for it, and finally authenticate to private docs if needed. The exact steps are shown in the Quick Start flow below.

Data storage and privacy

All data stays locally on your machine. The storage layout includes a SQLite database for metadata, a vector store for embeddings, and caches for crawled data.

Troubleshooting

If you encounter issues, re-index the documentation, verify the site renders with a headless browser, and check the local crawl data directory for clues.

Data paths and storage

All data is stored locally in your home directory under an MCP-specific folder. This includes the database, vectors, sessions, and crawled data.

Notes on authentication

For private docs, you can authenticate via an interactive browser login. The session is saved for future crawls, so you won’t need to re-authenticate each time.

Versioning and collections

Index multiple versions of the same package by providing a version tag during add_documentation. Use collections to group related docs for a project and search within a collection.

Available tools

add_documentation

Register a new documentation site for indexing, including optional tags, version, and authentication settings.

search_documentation

Perform a hybrid search over indexed docs with optional URL, tags, and result limit.

authenticate

Open a browser to log in to protected documentation and save the session for future crawls.

list_documentation

List all indexed documentation sites with metadata.

set_tags

Assign or update tags for a documentation site to aid filtering.

list_tags

Display all tags and their usage across indexed docs.

reindex_documentation

Re-crawl and refresh indexing for a specific site.

get_indexing_status

Check the current progress of ongoing indexing tasks.

delete_documentation

Remove an indexed site and its data from the local store.

clear_auth

Clear saved authentication sessions for a domain.

create_collection

Create a new collection to group related documentation sites.

add_to_collection

Add multiple documentation sites to a collection.

search_collection

Search within a specific collection using the same hybrid search.

list_collections

List all collections with their document counts.

get_collection

Retrieve details of a specific collection and its docs.

update_collection

Rename or update a collection's description.

remove_from_collection

Remove docs from a collection without deleting the docs themselves.

delete_collection

Delete a collection while keeping its indexed docs intact.