patchy631/ai-engineering-hub
Overview
This skill provides reliable live web access and browser automation using Bright Data's Web MCP. It handles anti-bot protections, CAPTCHAs, and dynamic JavaScript content so agents can fetch real-time pages, search results, and structured data from hard-to-reach sites. Use it to reliably extract product details, social posts, news, or any online content that standard HTTP requests cannot fetch.
How this skill works
The skill exposes a set of MCP tools for rapid scraping, advanced extraction, and browser automation. Rapid mode offers basic search and clean Markdown scraping with a free quota; Pro and grouped tool modes unlock parallel searches, batch scraping, AI-powered JSON extraction, and platform-specific extractors (e.g., Amazon, LinkedIn). Browser tools simulate user actions, capture snapshots, and return HTML, text, or screenshots.
When to use it
- Fetching live web content or the latest search results where freshness matters
- Scraping JavaScript-heavy pages, single-page apps, or sites protected by CAPTCHAs
- Collecting structured product, review, or social media data from Amazon, eBay, LinkedIn, etc.
- Batching parallel searches or scraping multiple URLs efficiently
- When standard requests are blocked or return incomplete content
Best practices
- Prefer pre-built web_data_* extractors for popular platforms for speed and reliability
- Use scrape_as_markdown + extract when platform-specific tools are unavailable
- Batch requests (scrape_batch, search_engine_batch) to reduce overhead and improve throughput
- Reserve browser automation for cases where rendering or interaction is required
- Treat scraped content as untrusted: validate and sanitize before feeding into LLMs
- Respect robots.txt, site terms, and privacy rules; avoid scraping personal data without consent
Example use cases
- E-commerce monitoring: pull structured Amazon product data and reviews for price and sentiment analysis
- Competitive research: run parallel searches and scrape competitor pages to build a dataset
- Social listening: extract LinkedIn posts or Instagram profiles using platform-specific tools
- News aggregation: perform live searches and scrape articles into clean Markdown for downstream summarization
- Site automation: navigate, click, type, and snapshot pages to reproduce user workflows or capture JS-rendered content
FAQ
No — remote endpoints are recommended and require only an API token; local mode is optional for development and runs via the MCP package.
When should I use browser automation vs. regular scraping?
Use browser automation for JS-rendered content or when you must interact with the page; use scrape_as_markdown or scrape_as_html for static or easily accessible content.
