web-scraper_skill

Overview

This skill fetches web page content and converts HTML into clean, readable Markdown. It extracts article content while stripping navigation, footers, and ads to deliver focused text suitable for reading or downstream processing. Use it when you need plain-text versions of web pages, or when built-in web fetchers fail due to environment restrictions.

How this skill works

The tool requests a URL, parses the returned HTML, and extracts main article elements such as headings, paragraphs, lists, and code blocks. It then maps HTML structures to Markdown syntax (headings, lists, bold/italic, fenced code) and removes common noise like navigation bars and advertisements. Options control request timeout, maximum output length, and raw-HTML output when needed.

When to use it

• You want a clean, Markdown version of an article for reading or note-taking.
• You need to extract textual content from a URL for analysis or summarization.
• The environment restricts the built-in WebFetch tool or other fetch methods fail.
• You are scraping multiple pages and want a simple script to run in parallel.
• A website returns HTML but you need to strip boilerplate (nav, footer, ads).

Best practices

• Increase --timeout for slow sites or large pages to avoid premature errors.
• Use --max-length to limit output size when processing many pages or long articles.
• If you get 403 responses, try changing the User-Agent or use browser automation for JS-heavy sites.
• Run multiple fetches in parallel with background processes when collecting many URLs.
• Verify target pages don’t require JavaScript rendering; the scraper does not run a browser.

Example use cases

• Fetch a single article: fetch and convert https://example.com/article to Markdown for archiving.
• Batch scrape: run multiple fetch processes in parallel to collect a list of news pages.
• Content extraction for NLP: produce plain text for summarization, topic modeling, or entity extraction.
• Troubleshooting fetch failures: increase timeout or switch to raw HTML to debug parsing issues.