afrexai-web-scraping-engine_skill

Overview

This skill is a complete web scraping and data extraction engine covering legal compliance, architecture, anti-detection, parsing, and production data pipelines. It provides practical decision rules, request and browser patterns, and operational best practices for building reliable, scalable scrapers in Python. Use it to design compliant, maintainable, and robust scraping systems at scale.

How this skill works

The skill inspects target sites for legal constraints (robots.txt, ToS, data classification) and produces an architecture brief that matches content rendering and anti-bot posture. It provides ready patterns for HTTP requests, browser automation (Playwright), parsing (JSON-LD, Open Graph, CSS/XPath, tables), pagination, proxy strategies, and monitoring/validation pipelines. It also gives anti-detection and escalation rules for Cloudflare and other WAFs.

When to use it

• Building a new scraper or choosing tools for a target site
• Assessing legal risk before starting a data collection project
• Scaling single-page scrapers into production pipelines
• Bypassing complex JavaScript rendering and anti-bot defenses safely
• Designing retries, rate limits, and monitoring for reliable operations

Best practices

• Start with a legal compliance check: robots.txt, ToS, PII and copyright
• Prefer official APIs when available; only scrape when justified
• Use structured sources first (JSON-LD, Open Graph, microdata) before CSS/XPath
• Implement retries, exponential backoff, random jitter and respect Retry-After
• Match headers and UA distributions to proxy geo and simulate natural navigation
• Validate, deduplicate, and version data before storage; monitor success rates

Example use cases

• Crawl e-commerce listings daily for price monitoring with pagination and dedupe
• Extract product metadata from JS-heavy SPAs using Playwright and JSON-LD
• Set up competitor monitoring with scheduled scrapers, proxy rotation, and alerts
• Migrate a manual scraping workflow to Scrapy/Crawlee with a production pipeline
• Recover tabular datasets from archived pages using pandas.read_html and validation