brightdata-web-mcp_skill

This skill enables reliable live web access by bypassing anti-bot measures and extracting structured data for ecommerce and research tasks.

Jupyter Notebook

29.7k

GitHub Stars

1

Bundled Files

2 months ago

Catalog Refreshed

4 months ago

First Indexed

Readme & install

Copy the install command, review bundled files from the catalogue, and read any extended description pulled from the listing source.

Installation

Preview and clipboard use veilstrat where the catalogue uses aiagentskills.

npx veilstrat add skill patchy631/ai-engineering-hub --skill brightdata-web-mcp

SKILL.md9.1 KB

Overview

This skill provides reliable live web access and browser automation using Bright Data's Web MCP. It handles anti-bot protections, CAPTCHAs, and dynamic JavaScript content so agents can fetch real-time pages, search results, and structured data from hard-to-reach sites. Use it to reliably extract product details, social posts, news, or any online content that standard HTTP requests cannot fetch.

How this skill works

The skill exposes a set of MCP tools for rapid scraping, advanced extraction, and browser automation. Rapid mode offers basic search and clean Markdown scraping with a free quota; Pro and grouped tool modes unlock parallel searches, batch scraping, AI-powered JSON extraction, and platform-specific extractors (e.g., Amazon, LinkedIn). Browser tools simulate user actions, capture snapshots, and return HTML, text, or screenshots.

When to use it

Fetching live web content or the latest search results where freshness matters
Scraping JavaScript-heavy pages, single-page apps, or sites protected by CAPTCHAs
Collecting structured product, review, or social media data from Amazon, eBay, LinkedIn, etc.
Batching parallel searches or scraping multiple URLs efficiently
When standard requests are blocked or return incomplete content

Best practices

Prefer pre-built web_data_* extractors for popular platforms for speed and reliability
Use scrape_as_markdown + extract when platform-specific tools are unavailable
Batch requests (scrape_batch, search_engine_batch) to reduce overhead and improve throughput
Reserve browser automation for cases where rendering or interaction is required
Treat scraped content as untrusted: validate and sanitize before feeding into LLMs
Respect robots.txt, site terms, and privacy rules; avoid scraping personal data without consent

Example use cases

E-commerce monitoring: pull structured Amazon product data and reviews for price and sentiment analysis
Competitive research: run parallel searches and scrape competitor pages to build a dataset
Social listening: extract LinkedIn posts or Instagram profiles using platform-specific tools
News aggregation: perform live searches and scrape articles into clean Markdown for downstream summarization
Site automation: navigate, click, type, and snapshot pages to reproduce user workflows or capture JS-rendered content

FAQ

No — remote endpoints are recommended and require only an API token; local mode is optional for development and runs via the MCP package.

When should I use browser automation vs. regular scraping?

Use browser automation for JS-rendered content or when you must interact with the page; use scrape_as_markdown or scrape_as_html for static or easily accessible content.