Web Scraper

支持爬取动态网站页面的mcp工具

typescript

1

GitHub Stars

typescript

Language

6 months ago

First Indexed

2 months ago

Catalog Refreshed

Documentation & install

Readme and setup notes from the catalogue, plus a client-ready config you can copy for your MCP host.

Installation

Add the following to your MCP client configuration file.

Configuration

View docs

{
  "mcpServers": {
    "naku111-mcpserver": {
      "command": "npx",
      "args": [
        "-y",
        "web-scraper-mcp-puppeteer",
        "mcpserver-puppeteer-nakiwang"
      ]
    }
  }
}

You run a Web Scraper MCP Server built on the Model Context Protocol to extract and export web content. It supports multiple export formats, smart content extraction, and flexible rule sets, making it easy to scrape dynamic pages, batch multiple URLs, and tailor requests with custom headers.

How to use

Start by launching the MCP server in your environment. You will interact with the server through an MCP client to run scraping tasks, create rule sets, and control domain headers. The server can export scraped content in Markdown, Text, HTML, or JSON, and it can fetch multiple URLs in parallel with optional Puppeteer rendering for dynamic pages.

Key capabilities you will use include scraping a single URL with a chosen format, creating reusable rule sets to target page sections, applying domain-specific headers to bypass login walls, and performing batch scraping for multiple pages. When you encounter sites with dynamic content, enable Puppeteer rendering to ensure you capture the rendered page.

How to install

Prerequisites: you need Node.js and npm installed on your system.

Install or clone the project dependencies and build the server as needed for your environment.

Install dependencies, build, and prepare to run the server.

Additional setup and usage notes

MCP can be run using a local stdio command or a remote HTTP endpoint. The available tooling includes creating rule sets, scraping a single URL, performing batch scrapes, and configuring domain headers.

If you need to render SPA content, set usePuppeteer to true to enable a headless browser. You can adjust the rule set to target specific page regions, exclude navigation elements, and avoid sections you don’t want to scrape.

Use a batch scrape to process multiple URLs in one operation and export each result in your preferred format.

Troubleshooting and tips

If a site requires authentication, set domain headers to include credentials such as Authorization tokens or cookies.

For pages that do not render content as expected, enable Puppeteer rendering to ensure the dynamic content is loaded before extraction.

Example workflow

Create a rule set to identify the title, content, and media blocks you care about.
Use the scraping tool to fetch a URL with the desired format and optional headers.
If needed, run a batch scrape for several URLs and export all results to Markdown for documentation.

Available tools

scrape_url

Scrapes a single webpage, exporting content in a selected format (markdown, text, html, json) with optional Puppeteer rendering and rule set applied.

create_rule_set

Creates a named rule set that defines how to extract title, content, links, and images from pages, along with any elements to exclude.

set_domain_headers

Configures domain-specific request headers to handle authentication or access restrictions.

batch_scrape

Performs scraping for multiple URLs in parallel and exports each result in a chosen format.