Web Crawler

This MCP server provides intelligent web crawling capabilities with browser rendering and AI-powered link extraction, enabling you to query and retrieve relevant links from complex, JavaScript-heavy web pages in real time.

How to use

You interact with this MCP server through an MCP client. Authenticate with GitHub, then invoke the webCrawl tool by supplying a target URL and a natural language query describing the links you want. The server renders the page in a headless browser, extracts links and metadata, analyzes them with AI, and returns a ranked list of relevant links along with explanations for why each result matters.

Typical usage patterns include: crawling a website to collect blog post links, finding product pages within a category, extracting API reference documentation links, or gathering repository links from a page. You can perform multiple queries over time and refine results by adjusting the natural language prompt.

How to install

Prerequisites you need installed on your machine: Node.js (for server scripts and tooling) and a modern web browser automation environment. You will also work with GitHub OAuth credentials to secure access.

Step by step, follow these commands to set up and run the MCP server locally or remotely:

# Install dependencies
npm install

# Start the local development server (example)
npx wrangler dev

Additional sections

Configuration focuses on authentication, access control, and how you connect clients to the remote MCP server. The server uses GitHub OAuth for authentication and restricts access to the webCrawl tool to specific GitHub usernames configured in the source code.

Security and access control are enforced through GitHub OAuth, with tokens managed by the Cloudflare OAuth provider. You can verify your connected status in your MCP client after completing the OAuth flow.

Example client connection options include using a remote HTTP SSE endpoint or running a local stdio-based MCP client that invokes the remote server URL. The SSE endpoint for real-time communication is typically hosted at a URL of the form https://web-crawler-mcp.<your-subdomain>.workers.dev/sse.

Troubleshooting tips include ensuring the OAuth callback URL matches your GitHub OAuth App settings, verifying that the targeted URL is accessible from your environment, and checking that your MCP client is configured with the correct SSE URL for the webCrawl tool.

Notes on usage with MCP clients

You can connect from various MCP clients that support SSH-like or SSE-based interactions. After authentication, you will discover the webCrawl tool in the client’s tool list. When using clients like Claude Desktop or Cursor, ensure you configure the MCP connection either through a direct remote command or via the provided SSE URL.

Available tools

webCrawl

Crawls a given URL and uses natural language queries to identify and rank relevant links. Returns matching links with explanations.

userInfoOctokit

Provides authenticated user information from GitHub, used for access control and profile data.