Gremlin Web Scraper

Gremlin Web Scraper MCP lets you run a lightweight Python-based HTTP MCP module locally to extract visible text from public webpages. It exposes simple endpoints for scraping single pages, crawling within a domain, and streaming results, all accessible through MCP clients in VS Code. This makes it easy to integrate web-text extraction into your workflows without needing external APIs or keys.

How to use

You interact with the Gremlin Web Scraper MCP using an MCP client in your development environment. Start the local server, then send requests to its endpoints to fetch and aggregate visible text from webpages.

Key capabilities you can use include scraping a single page, crawling a set of pages within the same domain, streaming results as pages are processed, and performing a health check. These endpoints are designed to return clean, plain text or structured mappings of URLs to their extracted text.

How to install

Prerequisites you need on your machine:

• Python 3.8+ installed on your system
• pip (package installer for Python) available in your PATH

Follow these steps to set up and run locally:

# Step 1: Ensure Python is installed
python --version
pip --version

# Step 2: Acquire the project files (clone or copy to your MCP workspace)
# Example using git (if available in your setup):
# git clone https://example.com/gremlin-mcp-scrap
# Or place the project under your MCP workspace directory

# Step 3: Install Python dependencies
pip install -r requirements.txt

# Step 4: Start the local MCP server
python server.py

Configuration and usage notes

This MCP is designed to run locally and exposes the following endpoints for MCP clients to call. Each endpoint returns plain JSON with the extracted text where applicable.

End points covered by this MCP:

• POST /scrape: fetches visible text from a single page URL.
• POST /crawl: recursively crawls links on the same domain up to configured limits.
• POST /crawl-stream: streams each fetched page as soon as it is retrieved.
• GET /ping: health check to verify the server is responsive.
• GET /mcp/metadata: discovery information about the MCP and its endpoints.

Examples of typical usage patterns

Send a request to scrape a single page and obtain the visible text as JSON.

curl -X POST http://localhost:8742/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com"}'

Request a recursive crawl limited to a few pages within the same domain.

curl -X POST http://localhost:8742/crawl \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","max_pages":10,"max_depth":2,"concurrency":5}'

Troubleshooting and tips

If the server is slow to respond or you encounter unusual pages, the MCP handles timeouts and errors gracefully. Ensure Python dependencies are installed and the server has network access. Check that the server is running and reachable at the expected host/port.

Notes

This MCP is designed to be simple to integrate into your MCP workflow. It returns clean JSON with extracted text, enabling downstream processing, storage, or analysis in your tooling.

Additional content

Security considerations: run the MCP on trusted networks and avoid exposing it publicly without proper access controls. For large-scale usage, consider deploying with appropriate rate limiting and monitoring.

Available tools

scrape

Fetch visible text from a single URL via POST /scrape and return the extracted text.

crawl

Recursively crawl links within the same domain via POST /crawl and return a mapping of URLs to extracted texts.

crawl_stream

Stream page-by-page text results via POST /crawl-stream as pages are fetched.

ping

Health check endpoint GET /ping that returns a simple status response.

mcp_metadata

Discover MCP metadata via GET /mcp/metadata.