data-scraper_skill

Overview

This skill provides a lightweight web data scraper that collects page content and extracts structured text without a browser. It supports readable text extraction, CSS selector targeting, table conversion, link harvesting, batch scraping, and change monitoring. The tool is optimized for command-line workflows and produces outputs in text, JSON, CSV, or Markdown.

How this skill works

The scraper fetches pages via HTTP using curl-style requests, then parses HTML to strip scripts and styles or to extract specific elements by CSS selector. Table mode maps HTML table headers to row objects, link mode normalizes and filters absolute URLs, and a watch mode snapshots content to detect diffs. It supports custom headers, cookies, user-agent overrides, rate limiting, and respectful robots.txt behavior when requested.

When to use it

• Extract readable article text for summarization or archiving
• Scrape product prices, listings, reviews, or ratings from e-commerce pages
• Convert HTML tables into JSON or CSV for data analysis
• Batch-collect data across many URLs with rate limiting
• Monitor pages for content or price changes and get alerted on diffs

Best practices

• Respect robots.txt and use the --polite flag for ethical scraping
• Set appropriate delays (--delay) and default rate limits to avoid 429s
• Use CSS selectors to narrow extraction and reduce noise
• Provide authentication headers or cookies for gated content
• Store snapshots and results in organized folders for reproducibility

Example use cases

• Fetch clean article text: data-scraper fetch URL --format md for downstream summarization
• Track a product price: data-scraper watch URL --selector ".price" --interval 3600 and alert on change
• Bulk export tables: data-scraper table URL --index 0 --format json to feed a data pipeline
• Crawl links for PDFs: data-scraper links URL --filter "*.pdf" to compile resources
• Batch scrape a list of pages with a delay: data-scraper batch urls.txt --delay 2000 --output results/