crawl4ai_skill
- Python
4
GitHub Stars
1
Bundled Files
2 months ago
Catalog Refreshed
4 months ago
First Indexed
Readme & install
Copy the install command, review bundled files from the catalogue, and read any extended description pulled from the listing source.
Installation
Preview and clipboard use veilstrat where the catalogue uses aiagentskills.
npx veilstrat add skill basher83/agent-auditor --skill crawl4ai- SKILL.md14.9 KB
Overview
This skill provides a production-ready web crawling and extraction toolkit based on Crawl4AI. It helps you scrape static and JavaScript-heavy pages, generate clean markdown, and extract structured data using reusable schemas for fast, LLM-free pipelines. The skill includes scripts and SDK patterns for batch crawling, session handling, and advanced crawl strategies.
How this skill works
The skill runs a headless browser to load pages, execute JavaScript, and capture HTML, markdown, media, and metadata. It offers configurable BrowserConfig and CrawlerRunConfig objects to control timeouts, selectors, JS hooks, screenshots, and session persistence. For structured extraction you can generate a reusable schema or supply CSS/JSON extraction definitions; the schema-based approach enables fast, deterministic extraction without LLM calls.
When to use it
- Scraping sites that require JS rendering or user interactions (infinite scroll, load-more).
- Converting documentation or knowledge bases to clean markdown for downstream use.
- Building repeatable pipelines for e-commerce product feeds or price monitoring.
- Crawling and aggregating multiple URLs concurrently with controlled concurrency.
- Extracting structured records where schema generation avoids per-run LLM costs.
Best practices
- Start with simple BrowserConfig and CrawlerRunConfig to validate selectors and timeouts.
- Generate a schema once for repetitive pages, then reuse it for fast extraction.
- Use wait_for and page_timeout to handle dynamic content; increase timeout for slow pages.
- Respect site rate limits: set max_concurrent and add short delays between requests.
- Persist sessions with session_id for login-required workflows instead of re-running login steps.
Example use cases
- Convert an entire docs site to markdown for LLM indexing and search.
- Generate a product schema from a shop, then run nightly extraction for price/availability monitoring.
- Crawl news sites concurrently and extract article bodies using fit-markdown filters.
- Automate login and crawl members-only pages by saving a session and reusing it.
- Run batch crawls of thousands of URLs with arun_many and controlled concurrency.
FAQ
Yes — use wait_for, js_code to run scrolling or clicks, and increase page_timeout; virtual_scroll_config supports complex virtual scrolling patterns.
When should I generate a schema instead of using an LLM?
Generate a schema when pages follow a repeatable structure. A schema yields much faster, cheaper extraction for recurring runs and eliminates LLM costs.
