firecrawl-skills_skill

Overview

This skill provides a command-line interface for web scraping, crawling, mapping, and full-text search optimized for LLM workflows. It returns clean Markdown (and other structured formats) with options for JavaScript rendering, site-mapping, and large-site crawls. Use it to extract readable content for research, documentation, competitive intelligence, or ongoing content monitoring.

How this skill works

firecrawl-cli runs searches and fetch jobs against web sources, optionally rendering JavaScript, bypassing common blocks, and extracting main content as Markdown, HTML, links, or JSON. It can perform single-page scrapes, full-site crawls with limits and depth controls, and URL discovery (map) with filters and subdomain options. Results are saved to a .firecrawl directory by default and are formatted to be LLM-friendly and easy to parse with jq, grep, or incremental reads.

When to use it

• Collect readable documentation or knowledge to feed into an LLM context window.
• Build a site map or discover all public URLs for inventory or audit purposes.
• Perform focused research or competitive intelligence across web, news, and GitHub.
• Crawl a website to archive content or extract bulk pages for analysis.
• Monitor web content changes or scrape pages with JavaScript-driven content.

Best practices

• Store outputs in a .firecrawl folder and add it to .gitignore to avoid repo bloat.
• Use -o to write results to files and avoid pasting large outputs directly into LLM prompts.
• Limit crawl depth and concurrency to respect site bandwidth and avoid rate limits.
• Preview large output files with head, wc, or grep instead of opening them fully.
• Use --wait and --progress for long-running crawls and check credit usage before large jobs.

Example use cases

• Scrape a product docs site into Markdown for building a model-specific knowledge base.
• Run a search with --scrape to gather top results and save cleaned content for analysis.
• Crawl a competitor site to map all public pages and extract changeable sections.
• Map a domain to create a URL inventory, then filter with --search to list blog posts.
• Parallel-scrape a list of URLs with xargs or shell background jobs for bulk extraction.