AI-Cursor-Scraping-Assistant

Provides an MCP server that analyzes pages, generates XPath selectors, and scaffolds Scrapy/Camoufox scrapers for e-commerce sites.

python

43

GitHub Stars

python

Language

6 months ago

First Indexed

2 months ago

Catalog Refreshed

Documentation & install

Readme and setup notes from the catalogue, plus a client-ready config you can copy for your MCP host.

Installation

Add the following to your MCP client configuration file.

Configuration

View docs

{
  "mcpServers": {
    "thewebscrapingclub-ai-cursor-scraping-assistant": {
      "command": "python",
      "args": [
        "xpath_server.py"
      ]
    }
  }
}

You use the MCP server to empower Cursor AI to analyze web pages, generate XPath selectors, and scaffold web scrapers with built-in support for Camoufox when needed. This server-based workflow helps you quickly transform websites into reusable scraping components while handling anti-bot considerations and modern page structures.

How to use

Start by opening your MCP-enabled client and connecting to the local MCP server you configured. Ensure the server is running and accessible from your Cursor environment. You can guide Cursor to analyze a target site and generate a complete scraper by issuing a prompt that describes the scraper you want, for example, a product listing page scraper for a given site. The system will analyze the website structure, check for anti-bot protections, extract relevant HTML elements, and produce a Scrapy-based spider aligned with the chosen scraper type.

How to install

Prerequisites you need before installation:

Python 3.10 or newer
pip (Python's package manager)
Cursor AI installed (or equivalent MCP client)

Step-by-step commands to set up the MCP server locally:

# 1) Clone the project repository (the MCP-enabled assistant)
git clone https://github.com/TheWebScrapingClub/AI-Cursor-Scraping-Assistant.git
cd AI-Cursor-Scraping-Assistant

# 2) Install required Python dependencies
pip install mcp camoufox scrapy

# 3) Fetch Camoufox browser binary if you plan to use Camoufox
python -m camoufox fetch

Configuration and operation notes

To run the MCP server locally, you will start the Python script that hosts the XPath/Camoufox tooling. Make sure you point the server to your local template if you customize the path.

Troubleshooting

If Cursor cannot connect to the MCP server, verify the server process is running and listening on the expected interface. Check that any local paths referenced by the server script are correct and accessible. If anti-bot protections block access, review your Camoufox configuration and ensure the template path is valid.

Notes

The MCP server is designed to work with Cursor AI to simplify scraper generation. It supports HTML analysis, extraction of XPath selectors, and optional Camoufox-based stealth scrapers. You can extend the browser rules and scraper templates to cover additional site types as your scraping needs grow.

Available tools

XPath selector generation

Create precise XPath expressions for target elements to drive scraper data extraction and element targeting.

Website analysis

Assess the structure of the target site, identify page types (PLP, PDP), and detect data sources such as JSON data and schema.org markup.

Anti-bot protection detection

Identify bot protection mechanisms (Akamai, Datadome, PerimeterX, etc.) and adapt scraping strategies accordingly.

Camoufox integration

Utilize Camoufox templates to fetch content and produce stealth scrapers that bypass basic anti-bot measures.

JSON data extraction

Identify and extract embedded JSON data within pages to populate structured fields in scrapers.

Schema.org/modern framework detection

Detect schema.org markup and frameworks like Next.js to improve data mapping and page understanding.