Website Scraper

You can access the Website Scraper MCP via authenticated Bearer token access or through paid HTTP 402 flows, enabling AI agents to interact with Website Scraper MCP endpoints through standardized tools. This server bridges your MCP clients to the Website Scraper MCP API with secure authentication and flexible deployment options.

How to use

Connect your MCP client to the server using the MCP URL shown in the deployment notes. You can operate in two modes: authenticated access using your own Website Scraper MCP API key, or paid access via the HTTP 402 protocol. In either mode, the server exposes tools under the /mcp endpoint that your client can call through the MCP adapter. Start by configuring your client to point at the server URL, then list available tools and invoke them with your desired arguments.

How to install

Prerequisites: Docker and Docker Compose (recommended) or Python with uvw dependencies if you prefer manual startup.

Using Docker (Recommended) Start by cloning the project, then set up your environment and run the docker workflow.

# Clone the repository
git clone https://github.com/Traia-IO/website-scraper-mcp-mcp-server.git
cd website-scraper-mcp-mcp-server

# Set your API key for Website Scraper MCP access (used by the server)
export WEB_SCRAPPING_API_KEY="your-api-key-here"

# Start the server with Docker
./run_local_docker.sh

Additional setup and usage notes

If you prefer Docker Compose, create a .env file with your configuration, then start the server with docker-compose up. The setup includes optional payment mode parameters and operator keys for on-chain settlement.

# Server's internal API key (for payment mode)
WEB_SCRAPPING_API_KEY=your-api-key-here

# Server payment address (for HTTP 402 protocol)
SERVER_ADDRESS=0x1234567890123456789012345678901234567890

# Operator keys (for signing settlement attestations)
MCP_OPERATOR_PRIVATE_KEY=0x1234567890abcdef...
MCP_OPERATOR_ADDRESS=0x9876543210fedcba...

# Optional: Testing mode (skip settlement for local dev)
D402_TESTING_MODE=false
PORT=8000

Usage with an MCP client

To use the server from a client, connect via the MCP adapter, authenticate if you have an API key, and enumerate the available tools. Then call a tool with your desired arguments and handle the response.

Available tools

example_tool

Placeholder tool (to be implemented)