doc-scraper_skill

Overview

This skill scrapes documentation websites and converts them into organized, categorized reference files suitable for searchable archives or Claude skills. It focuses on reliable extraction, category mapping, checkpointed runs, and safe rate-limited crawling. The skill outputs a tidy directory of reference pages plus raw JSON and summary metadata for downstream processing.

How this skill works

First, the scraper validates the target: URL accessibility, robots.txt, selector viability, and writable output path. Next it generates a config with base_url, CSS selectors, include/exclude URL patterns, categories, and rate limits. The scraper crawls pages (sitemap or crawl start), extracts main content and code blocks per selectors, categorizes pages, saves structured markdown and raw JSON, and records checkpoints for resuming interrupted runs.

When to use it

• Converting a docs site into local, searchable reference files for an LLM skill
• Building offline archives or knowledge bases from API docs or guides
• Standardizing documentation into categorized markdown for developer portals
• Preparing source material for agentic workflows or Claude skill training
• Migrating docs content while preserving code samples and structure

Best practices

• Inspect the site manually before running: test curl -I and check robots.txt
• Define precise selectors for main content, titles, and code blocks to avoid noise
• Use include/exclude URL patterns and category keywords to reduce misclassification
• Respect rate limits and start with low concurrency; increase only after testing
• Enable checkpoints for large crawls and verify output structure before bulk retries

Example use cases

• Scrape a library's API docs into categorized markdown for offline reference
• Harvest tutorial and guide pages to bootstrap a Claude skill knowledge base
• Create a searchable developer handbook from scattered docs and how-tos
• Migrate multiple versioned docs into a single standardized folder layout
• Extract code examples and save raw JSON for downstream analysis or testing