MCP Web Scrape

You can leverage MCP Web Scrape to reliably obtain clean, agent-ready content from the web with automatic citations, robust caching, and robots-friendly behavior. It exposes a straightforward way to extract, transform, and analyze web data for use in your MCP workflows and AI agents.

How to use

Start the MCP server and use it from your MCP client by invoking the local or global command and then sending requests through your agent prompts. The tool focuses on converting web content into clean Markdown, extracting structure and metadata, and providing ready-to-use data for downstream tasks.

How to install

Prerequisites you need before installing:

• Node.js installed on your system (version 14 or newer)

• npm (comes with Node.js) or yarn if you prefer it

Install and run commands

# Install specific version globally
npm install -g mcp-web-scrape@1.0.7

# Or try it instantly (latest version)
npx mcp-web-scrape@latest

# Try specific version
npx mcp-web-scrape@1.0.7

# Or start HTTP server
node dist/http.js

Using with a client (stdio transport)

You can run the server locally via a standard input/output (stdio) transport. The command and arguments shown below are the actual runtime instructions you would use.

{
  "mcpServers": {
    "web_scrape": {
      "type": "stdio",
      "command": "npx",
      "args": ["mcp-web-scrape@1.0.7"]
    }
  }
}

Accessing cached content and data

Once running, you can access cached content and other MCP resources using the built-in cache system. Examples include reading a cached page, viewing cache statistics, and checking robots.txt status.

Available tools

extract_content

Convert HTML to clean Markdown with citations

summarize_content

AI-powered content summarization

get_page_metadata

Extract title, description, author, keywords

extract_links

Get all links with filtering options

extract_images

Extract images with alt text and dimensions

search_content

Search within page content

check_url_status

Verify URL accessibility

validate_robots

Check robots.txt compliance

extract_structured_data

Parse JSON-LD, microdata, RDFa

compare_content

Compare two pages for changes

batch_extract

Process multiple URLs efficiently

get_cache_stats

View cache performance metrics

clear_cache

Manage cached content

extract_forms

Extract form elements, fields, and validation rules

extract_tables

Parse HTML tables with headers and structured data

extract_social_media

Find social media links and profiles

extract_contact_info

Discover emails, phone numbers, and addresses

extract_headings

Analyze heading structure for content hierarchy

extract_feeds

Discover and parse RSS/Atom feeds

convert_to_pdf

Convert web pages to PDF format with customizable settings

extract_text_only

Extract plain text content without formatting or HTML

generate_word_cloud

Generate word frequency analysis and word cloud data

translate_content

Translate web page content to different languages

extract_keywords

Extract important keywords and phrases from content

analyze_readability

Analyze text readability using metrics like Flesch

detect_language

Detect the primary language of web page content

extract_entities

Extract named entities (people, places, organizations)

sentiment_analysis

Analyze sentiment and emotional tone of content

classify_content

Classify content into categories and topics

analyze_competitors

Analyze competitor websites for SEO and content insights

extract_schema_markup

Extract and validate schema.org structured data

check_broken_links

Check for broken links and redirects on pages

analyze_page_speed

Analyze page loading speed and performance metrics

generate_meta_tags

Generate optimized meta tags for SEO

scan_vulnerabilities

Scan pages for common security vulnerabilities

check_ssl_certificate

Check SSL certificate validity and security details

analyze_cookies

Analyze cookies and tracking mechanisms

detect_tracking

Detect tracking scripts and privacy concerns

check_privacy_policy

Analyze privacy policy compliance and coverage

monitor_uptime

Monitor website uptime and availability

track_changes_detailed

Advanced change tracking with similarity analysis

analyze_traffic_patterns

Analyze website traffic patterns and trends

benchmark_performance

Benchmark performance against competitors

generate_reports

Generate comprehensive analysis reports

monitor_changes

Track content changes over time with similarity analysis

analyze_performance

Measure page performance, SEO, and accessibility metrics

generate_sitemap

Crawl websites to generate comprehensive sitemaps

validate_html

Validate HTML structure, accessibility, and SEO compliance