firecrawl-crawl_skill

Overview

This skill bulk-extracts content from an entire website or a specific site section. It runs a link-following crawl with configurable depth, page limits, path filters, and concurrent workers. Use it to collect many pages for analysis, ingestion, or offline processing. Results can be returned synchronously or as an async job ID for polling.

How this skill works

The crawler starts from a seed URL and follows internal links up to a maximum depth and page limit. You can include or exclude path patterns to scope the crawl, set request delays, and tune concurrency for performance and politeness. The skill outputs a JSON file containing scraped page content and metadata, or returns a job ID when run asynchronously. Progress and wait options let you monitor or block until completion.

When to use it

• You need content from many pages on the same site (e.g., an entire /docs section).
• You want to extract every page under a specific path for indexing or archival.
• You need to bulk-scrape linked pages rather than single-page scrapes.
• You want controlled breadth/depth with concurrency and delay settings for large crawls.

Best practices

• Scope the crawl with --include-paths to avoid crawling the whole site and reduce cost.
• Use --limit and --max-depth to control crawl size and runtime.
• Set --delay and reasonable --max-concurrency to avoid overloading the target site.
• Run with --wait when you need immediate results; otherwise poll the returned job ID.
• Check credit usage before large crawls to manage costs.

Example use cases

• Crawl and extract all developer docs under /docs for building a searchable knowledge base.
• Bulk-extract product pages across a site section for competitive analysis or feed generation.
• Crawl a blog archive to ingest posts into an AI training or summarization pipeline.
• Run a depth-limited crawl to map and scrape linked help articles for support automation.