x-scraper_skill

Overview

This skill scrapes X (Twitter) content via a browser connected through Chrome DevTools Protocol (agent-browser). It provides two scripts to harvest a user timeline or the logged-in home recommendation stream and exports results as Markdown, RSS, or JSON. The tool supports keyword filtering, tab switching (For you / Following), and configurable scrolling limits.

How this skill works

The scripts control a CDP-connected browser to navigate X, load timelines, and scroll to collect visible posts. scrape_x_user.sh targets a specific username and can filter posts by keyword; scrape_x_home.sh captures the logged-in account’s home feed and can switch between recommendation tabs. Output includes structured fields (author, time, text, link, media flag, retweet context) and is written to stdout or a file in md/rss/json.

When to use it

• Harvest a specific user’s recent tweets for archiving or analysis.
• Monitor a public figure or project feed and filter by keywords or topics.
• Capture the logged-in account’s For You or Following recommendation stream for research.
• Generate RSS or JSON feeds from X content for downstream consumption.
• Quickly export timelines to Markdown for note-taking or reporting.

Best practices

• Ensure agent-browser CDP is running and the browser is logged into X before running scripts.
• Use modest max scroll counts for initial runs to avoid long sessions and to validate results.
• Combine keyword filters with user scraping to reduce noise and focus on relevant posts.
• Prefer JSON output for programmatic processing; use RSS for feed consumption and Markdown for human review.
• Rotate output file paths when collecting repeated snapshots to avoid overwriting historical captures.

Example use cases

• Daily scrape of a project lead’s timeline, filtering for release-related keywords, output to JSON for downstream analytics.
• Capture your logged-in For You feed weekly to study recommendation differences versus Following, export as RSS for review.
• Create a Markdown digest of a researcher’s posts filtered by topic for inclusion in a weekly newsletter.
• Fetch 10 pages of a user timeline to build a local archive and convert to JSON for import into a search index.
• Quickly generate an RSS feed from the home recommendation stream for a private monitoring dashboard.