MCP Web Scraper

You can run the mcp-web-scraper as a local MCP server that uses Google Chrome in headless mode to scrape web pages for AI agents. It works well with JavaScript-heavy sites and can be integrated into your MCP workflow or used through LangChainGo for Go-based tooling.

How to use

Start by compiling the server, then configure your MCP client to connect to it as a local (stdio) server. When Chrome is available, the scraper uses headless Chrome to render pages. If Chrome isn’t installed, it falls back to a plain Go HTTP user agent, which still works for many sites that don’t require JavaScript. When content is returned, the module attempts to convert articles to Markdown by default, and will strip excessive HTML (like <script> and <svg> tags and comments) to reduce the token load for the LLM. You can rely on Markdown when the page presents itself as an article; otherwise you’ll receive HTML content that’s cleaned up for easier processing.

How to install

Prerequisites: you need Go installed on your machine to build the server.

Step-by-step build and run flow:

• Go to your project directory where the source resides.

• Compile the server.

• Configure your MCP client to connect to the local server using the provided command and arguments.

• Start the server and verify it’s reachable by your MCP client.

Examples and configuration

{
  "mcpServers": {
    "mcp-web-scraper": {
      "command": "/path/to/mcp-web-scraper",
      "args": []
    }
  }
}

Additional notes

If you want to use the server from your Visual Studio Code MCP integration, configure the same server entry in your VS Code MCP settings. The command remains the same and you can pass an empty arguments array if you don’t need extra flags.

LangChainGo tool

Integration into LangChainGo is straightforward. You can instantiate a scraper using the LangChainGo helper.

Fallback modes

If Google Chrome is not installed, the scraper will fall back to using Go's HTTP user agent. This works for many sites, but you may not receive content rendered by JavaScript.

Markdown support

By default, this module looks for an <article> element and converts it to Markdown. If the page doesn’t present content as an article, it returns HTML instead. Returned HTML will have elements like <script>, <svg>, and comments removed to help reduce token usage for the LLM.

Available tools

langchain_scraper

Go LangChain integration to create and use a web scraper tool within LangChainGo.

md_article_converter

Converts HTML articles to Markdown by default and strips non-essential HTML elements to reduce token usage.