scrapedo-web-scraper_skill

Overview

This skill integrates Scrape.do as a fallback web scraper for pages that fail with standard HTTP fetches. It bypasses common blocks like Cloudflare, JavaScript rendering barriers, and simple CAPTCHAs to return page text or full HTML. It triggers automatically on common WebFetch errors to preserve workflow continuity.

How this skill works

The skill routes requests to the Scrape.do API when WebFetch returns 403, 401, 429, timeout, access denied, or Cloudflare block. It performs headless rendering and anti-block measures, then returns either cleaned text or raw HTML depending on the options. On API errors it returns a clear, human-readable error explaining token, rate limit, or availability issues.

When to use it

• Standard fetch returns 403, 401, 429, timeout, or access denied
• Pages blocked by Cloudflare or similar anti-bot services
• Pages that require JavaScript rendering to load content
• When you need raw HTML for parsing or the rendered text for NLP
• As an automatic fallback in production scraping pipelines

Best practices

• Keep the Scrape.do API token secure and rotate it per your security policy
• Use HTML output only when you need full markup; prefer text for downstream NLP
• Respect robots.txt and site terms; use rate limiting and backoff to avoid bans
• Handle returned error messages programmatically to fall back or alert on quota issues
• Cache results for stable pages to reduce API usage and costs

Example use cases

• Automated content extraction when a site blocks direct fetches with 403/Cloudflare
• Collecting article text from dynamic sites that require JS rendering
• Fallback layer in a scraping service to reduce failed fetches and false negatives
• Getting raw HTML for complex parsers that rely on client-side rendering