bright-data_skill

Overview

This skill provides ready-to-use curl examples and guidance for the Bright Data Web Scraper API, focusing on social media scraping, account checks, and usage monitoring. It helps you trigger synchronous or asynchronous scrapes, track job progress, download snapshots, and manage account/zone info using simple shell commands. Designed for operators who prefer direct API calls and scripting with environment variables.

How this skill works

You send JSON payloads to Bright Data dataset endpoints to start scraping jobs (either async via /trigger or sync via /scrape). The API returns a snapshot_id for async runs; poll /progress until status is ready, then download results from the snapshot endpoint. Additional endpoints provide account status, active zones, and bandwidth usage. All examples use curl and expect an exported BRIGHTDATA_API_KEY environment variable.

When to use it

• Collect posts, profiles, comments, and engagement metrics from Twitter/X, Reddit, YouTube, Instagram, TikTok, or LinkedIn.
• Run large batch collections or scheduled discovery jobs (use async /trigger).
• Perform quick lookups for single pages or videos (use sync /scrape).
• Monitor account health, active zones, and bandwidth usage programmatically.
• Integrate scraping into shell scripts, cron jobs, or CI pipelines that require simple curl calls.

Best practices

• Create and select a Web Scraper dataset in the Bright Data Control Panel to obtain your dataset_id before calling the API.
• Export BRIGHTDATA_API_KEY and wrap curl commands in bash -c '...' when piping to avoid environment-variable clearing issues.
• Use /trigger for large or multi-URL jobs and poll /progress until status is ready before downloading snapshots.
• Respect rate limits (max ~100 concurrent requests and 1GB input size per batch) and handle HTTP 429 gracefully with retries/backoff.
• Filter results with parameters like num_of_posts, start_date/end_date (MM-DD-YYYY), and sort_by where supported to reduce costs and data volume.

Example use cases

• Trigger a scheduled crawl of a list of Twitter profiles and download consolidated JSON snapshots for analytics.
• Scrape YouTube video metadata and transcripts synchronously for a small set of video IDs.
• Collect subreddit posts and nested comments for sentiment analysis using /trigger then download when ready.
• Check account status, list active zones, and pull bandwidth usage daily to track consumption and costs.
• Fetch recent Instagram posts for a brand within a date range using date filters to limit results.