scraping_skill

Overview

This skill implements a four-tier progressive URL scraping and multi-page crawling pipeline that escalates from lightweight fetches to a professional proxy service. It reliably retrieves page content and site maps while handling bot detection, JavaScript rendering, and CAPTCHAs. Use it to get HTML or markdown outputs when simple requests fail or when you need a full-site crawl.

How this skill works

Before any scraping action the skill sends a required local voice notification and a short text notification announcing the workflow. It then routes requests to one of two workflows: a Four-Tier Scrape for single-URL retrieval or a Crawl workflow for multi-page site extraction. The four tiers run in order: WebFetch (fast built-in), curl with Chrome-like headers, Playwright browser automation, and Bright Data MCP (proxy/CAPTCHA resolver), automatically falling back only when a tier fails.

When to use it

• Fetch a single URL and convert it to markdown or raw HTML
• Site blocks requests, returns 403, or shows bot detection/CAPTCHA
• Pages require JavaScript rendering or dynamic content loading
• Crawl multiple pages or map a website section (light or full crawl)
• You need reliable extraction when standard fetch methods fail

Best practices

• Declare explicit scope and max pages when requesting crawls to limit cost
• Provide the target URL plus any login or cookie details if available
• Prefer progressive escalation; request direct Bright Data only when necessary
• Specify output format (markdown, HTML, or structured extraction) up front
• Check for local user customizations in the BrightData preferences path before running

Example use cases

• Quickly scrape a public blog post into markdown via WebFetch
• Fetch a JavaScript-heavy dashboard using Playwright when curl fails
• Bypass advanced bot detection and CAPTCHA using Bright Data MCP for protected pages
• Light crawl (<50 pages) to extract documentation and produce a site map
• Trigger a full-site Bright Data crawl for comprehensive indexing of a large domain