Scraper

This MCP server is a context-optimized web scraping engine that runs as a remote service to reduce your LLM token usage. It filters and converts HTML on the server, applies CSS selector targeting, and can render JavaScript when needed, delivering concise content tailored for large language models and automated tools.

How to use

You connect to the scraper MCP server from your client or workflow and issue scrape requests. The server exposes an MCP endpoint at the path /mcp and a dashboard at /. You can integrate it with Claude Code by adding the MCP as an HTTP transport and pointing to the server URL, for example http://localhost:8000/mcp. Use the available tools to request HTML converted to markdown, raw HTML, plain text, or links with metadata. When you enable JavaScript rendering, you can handle dynamic content from SPAs.

Typical usage patterns include:

Endpoints you will use frequently:

MCP endpoint: http://localhost:8000/mcp

Dashboard: http://localhost:8000/

How to install

Prerequisites: you need Docker or Docker Compose installed on your system. You may also run locally via the provided CLI options if you have a compatible runtime, but the recommended start is to run the official container.

Install and run with Docker (recommended):

# Run the MCP server in the background and expose port 8000
docker run -d -p 8000:8000 --name scraper-mcp ghcr.io/cotdp/scraper-mcp:latest

# Optional: if you plan to use JavaScript rendering, ensure you have enough memory available
# and set appropriate environment options when starting the container

If you prefer Docker Compose for persistence and configuration management, you can use a setup like this (adjust environment variables as needed):

# docker-compose.yml
services:
  scraper-mcp:
    image: ghcr.io/cotdp/scraper-mcp:latest
    ports:
      - "8000:8000"
    volumes:
      - cache:/app/cache
    environment:
      - PERPLEXITY_API_KEY=${PERPLEXITY_API_KEY:-}
      - PLAYWRIGHT_MAX_CONTEXTS=5
    deploy:
      resources:
        limits:
          memory: 1G  # Recommended for JS rendering
    restart: unless-stopped
volumes:
  cache:

Configuration and notes

Resource controls and feature flags can be enabled to tune performance and capabilities. Enable resources and prompts selectively to reduce context overhead. You can configure JavaScript rendering memory and concurrency, proxy settings, and optional integrations like Perplexity for AI-powered search and reasoning.

Sample environment variables to consider setting in your deployment: PERPLEXITY_API_KEY, PLAYWRIGHT_MAX_CONTEXTS, PLAYWRIGHT_TIMEOUT, PLAYWRIGHT_DISABLE_GPU, ENABLE_RESOURCES, ENABLE_PROMPTS, HTTP_PROXY, HTTPS_PROXY, SCRAPEOPS_API_KEY, SCRAPEOPS_RENDER_JS.

Examples and quick starts

Try a basic scrape in your workflow or CLI by sending a URL to the MCP server and requesting Markdown output suitable for LLMs. You can also combine rendering with a CSS selector to target a specific portion of a page.

# Example: basic scrape (Markdown output) via the HTTP MCP endpoint
curl -s http://localhost:8000/mcp -d '{"tool":"scrape_url","args":["https://example.com"]}'

Available tools

scrape_url

Convert HTML to Markdown output suitable for large language models.

scrape_url_html

Retrieve raw HTML content from the target page.

scrape_url_text

Extract plain text content from the page.

scrape_extract_links

Extract all links with metadata from a page.

perplexity

AI web search with citations using Perplexity.

perplexity_reason

Multi-step reasoning tasks using Perplexity.