documentation-scraper_skill

Overview

This skill scrapes documentation websites and compiles them into a single markdown file optimized for AI agent context. It is lightweight, deterministic, and does not use AI to perform scraping — it prepares source material for AI consumption. Always analyze the sitemap before scraping to set correct scope and limits. All scraping commands must be executed outside the sandbox environment.

How this skill works

The skill uses the slurp-ai CLI to fetch pages into partial markdown files and then compiles them into one consolidated document. It begins by running an included sitemap analysis script to report page counts, section grouping, and suggested --base-path and --max values. After verifying slurp-ai is installed, you run slurp with parameters informed by the sitemap, then compile partials into slurp_compiled/compiled_docs.md for AI context use.

When to use it

• You need a local, single-file markdown copy of online docs for an AI agent
• Preparing documentation as context for code assistants or knowledge bases
• Creating offline snapshots of versioned library docs
• Scoping large doc sites to a relevant subsection before scraping
• Automating documentation downloads for tooling or research

Best practices

• Always run node analyze-sitemap.js <site> first to determine total pages and sensible --base-path and --max values
• Verify slurp-ai is installed with which slurp before attempting any network commands
• Run all shell commands outside the sandbox (dangerouslyDisableSandbox: true) because network and file operations are blocked inside the sandbox
• Start with conservative --max and iterate after inspecting slurp_partials; increase only when confident
• Use --base-path to restrict crawl to the relevant section and --remove-duplicates when compiling

Example use cases

• Scrape the /docs/guides section of a library site after sitemap shows ~60 pages
• Download API reference pages only by setting --base-path to /api/ and --max to sitemap count
• Create a compiled markdown file of React or Express docs for local AI-assisted code exploration
• Automate nightly snapshots of a stable docs subsection for offline access
• Scope a large site (200+ pages) to specific sections identified by sitemap analysis