Better Fetch

Better Fetch is a powerful MCP server that intelligently fetches and processes web content with nested URL crawling capabilities. It converts complex web resources into clean, structured Markdown documents suitable for AI analysis and knowledge extraction, while offering flexible configuration to control scope and depth of crawling.

How to use

You will integrate Better Fetch with your MCP client to start crawling and transforming web content into well-structured Markdown. Begin by ensuring your client is configured to load an MCP server named better-fetch, then invoke the server’s fetch tools through your client’s command interface. Use nested crawling to automatically follow links within defined patterns and depths, or switch to single-page mode for focused extractions. The server cleans content, preserves code blocks and links, and attaches metadata like source URLs and timestamps to the generated Markdown.

How to install

Prerequisites include Node.js version 18 or newer and a package manager such as npm or yarn. You will also need an MCP-compatible client such as Claude Desktop or the VS Code MCP extension.

# Step 1: Clone the project
git clone https://github.com/flutterninja9/better-fetch.git
cd better-fetch

# Step 2: Install dependencies
npm install

# Step 3: Build the project
npm run build

# Step 4: (Optional) Run tests or start in development mode
npm run dev
node test-mcp.js

Configuration and client setup

Configure your MCP client to load the Better Fetch server through an MCP entry point. The recommended setup for Claude Desktop and for the VS Code MCP Extension uses npx to run the server on demand. You will add a server definition named better-fetch in your client configuration.

{
  "mcpServers": {
    "better-fetch": {
      "command": "npx",
      "args": ["-y", "@infero.mcp/better-fetch"]
    }
  }
}

Notes on startup and local runs

If you prefer manual startup without using npx, you can run the built server directly after building. The alternative configuration shows how to point a client at a local Node.js server instance and provide necessary environment variables as needed by your setup.

Security and best practices

Respectful crawling is enabled by default with delays between requests. Adjust crawl depth, page limits, timeouts, and pattern filters to minimize load on target sites and to focus on relevant content. Handle errors gracefully so that failed fetches do not interrupt your overall document generation workflow.

Examples and troubleshooting

Common workflows include extracting API documentation, framework docs, or tutorials into organized Markdown with a structured table of contents. If you encounter timeouts or blocked requests, increase the timeout, widen domain allowances, or apply include/exclude patterns to narrow down the crawl scope.

Tools and capabilities

The server provides functions to crawl and convert web pages into Markdown with metadata. Two core tools enable this behavior: fetch_website_nested for comprehensive, nested crawling, and fetch_website_single for simple, single-page extractions.

Maintenance and updates

Keep your client configuration up to date with the latest package version by using the standard package manager workflow (for example, updating dependencies via npm). Regularly review included crawl patterns and depth limits to align with evolving content needs.

Available tools

fetch_website_nested

Comprehensive web crawling with nested URL processing, including depth control, domain filtering, and pattern matching.

fetch_website_single

Simple single-page content extraction for focused captures without crawling additional links.