data-base_skill

Overview

This skill performs data acquisition for web scraping and structured data collection. It converts unstructured web pages into JSON or CSV outputs ready for analysis. Use it when you need reliable scraping workflows for both static and JavaScript-driven sites.

How this skill works

I ask what data and sites you need, then choose the appropriate tool based on page complexity: static pages use HTTP requests and HTML parsing, dynamic pages use a Chrome DevTools-backed browser controller, and complex pipelines run as Python scripts. The extractor respects robots.txt, rate limits, and a configurable User-Agent, then outputs {source}_{timestamp}.json or .csv with ASCII-only filenames.

When to use it

• You need tabular exports (CSV) or nested JSON from web pages
• Target pages are JavaScript-heavy or single-page applications
• You want reproducible, scriptable scraping with rate limits and proxies
• You need quick one-off scrapes of static sites like blogs or news
• You plan to feed scraped output into downstream analysis or ETL

Best practices

• Always check and respect robots.txt before scraping a site
• Default to conservative rate limiting (example: 1 request/second)
• Identify the scraper with a clear User-Agent string
• Avoid collecting personal data without consent and follow privacy rules
• Use ASCII-only filenames and timestamped outputs (source_YYYYMMDDHHMM.ext)

Example use cases

• Scrape article metadata (title, author, date, tags) from a news site into CSV
• Extract product listings and prices from a React e-commerce SPA using Chrome DevTools control
• Run a repeatable Python pipeline to gather paginated search results with retries and proxies
• Produce nested JSON records from forum threads for topic modeling
• Generate a timestamped dataset to feed into a data analysis notebook