fetch-url_skill

Overview

This skill fetch-url fetches a web page and extracts its main text content (default output: Markdown). It includes special handling for X/Twitter URLs to improve retrieval success on restricted pages. The tool runs as a standalone executable script and supports multiple output formats and fetch strategies.

How this skill works

Run the script directly from its directory as an executable to navigate the URL with Playwright or fall back to content readers. It can detect a local Chromium-like browser automatically or use Playwright-installed browsers, call Jina Reader when configured, or prefer raw site Markdown depending on the chosen fetch strategy. The script then extracts rendered content (optionally post-processed with trafilatura) and writes output to stdout or a file in the requested format.

When to use it

• Quickly capture and save the main text of any http/https page as Markdown or other formats.
• Fetch content from X/Twitter posts with improved handling for restricted or dynamically rendered pages.
• Automate content collection for offline reading, archiving, or downstream NLP processing.
• Prefer browser rendering for pages that require JS, or use Jina/agent modes for lighter, API-driven extraction.
• Produce raw rendered HTML when you need the exact DOM output rather than cleaned text.

Best practices

• Always run the script as an executable from its containing directory (do not invoke via python interpreter wrappers).
• Install Playwright browsers if no local Chromium is detected (example: uv run playwright install chromium).
• Set JINA_API_KEY in the environment when using the jina fetch strategy to reduce rate limits and improve reliability.
• Choose --fetch-strategy based on the page: browser for heavy JS sites, agent or jina for API-based extraction, auto for sensible defaults.
• Use --timeout-ms to increase navigation time for slow pages or complex renderings.

Example use cases

• Save a long-form article as Markdown for offline reading: ./scripts/fetch_url.py https://example.com --output ./page.md
• Extract a Twitter/X post as Markdown with browser rendering for dynamic content.
• Batch-crawl and store HTML or CSV outputs for downstream data processing or ingestion.
• Use the jina strategy with JINA_API_KEY for API-native extraction where available.