doc-scraper_skill

Overview

This skill scrapes sections of docs.snowflake.com and converts pages to organized Markdown files with frontmatter. It uses a local SQLite cache with a configurable expiration to avoid re-downloading unchanged pages. The spider depth and base path are configurable to control crawl scope and volume.

How this skill works

Run the scraper with an output directory and optional flags to set the base path and spider depth. The tool crawls allowed paths, follows links up to the specified depth, converts pages to Markdown, and stores metadata. A SQLite cache stores fetched pages for the configured expiration period to speed repeated runs and reduce load. Rate-limiting and thread settings control concurrency.

When to use it

• When you need an offline, searchable copy of Snowflake documentation sections.
• To build documentation sites, knowledge bases, or data platform playbooks from Snowflake docs.
• When auditing or tracking changes to specific Snowflake docs over time.
• To extract docs for inclusion in internal developer portals or training materials.

Best practices

• Set --base-path to limit the crawl to only the documentation area you need.
• Start with --spider-depth=0 or 1 to verify scope before increasing depth.
• Use the --limit flag for testing large sections to avoid long runs.
• Inspect and tune scraper_config.yaml for max threads and allowed paths to balance speed and site load.
• Use --dry-run to preview which pages will be processed before writing files.
• Remove the .cache directory only if you suspect cache corruption; otherwise rely on expiration_days to refresh content.

Example use cases

• Export the SQL reference section to Markdown for integration into an internal docs site using --base-path="/en/sql-reference/".
• Create a weekly snapshot of a migrations guide by running the scraper with caching and storing output in a version control repo.
• Generate a trimmed, offline copy of admin and security docs by adjusting allowed_paths and spider depth.
• Test new scraping rules on a small set using --limit and --dry-run before full extraction.