serp-scraper_skill

Overview

This skill scrapes Google Search Engine Results Pages (SERP) for SEO analysis, keyword tracking, and competitive intelligence. It extracts organic results, featured snippets, People Also Ask, ads, local pack entries, and other SERP features while using residential proxy rotation and a browser automation tool for reliability. The skill focuses on geo-accurate results and safe request pacing to avoid Google’s bot defenses.

How this skill works

It uses a browser tool configured with residential proxies to load rendered Google pages, accept cookie consent when required, and extract DOM elements for all visible SERP features. Requests are performed with slow, randomized delays and rotating country-targeted proxies to emulate real users and reduce CAPTCHA and blocking. Results are returned as structured JSON including timestamps, country/language, organic results, snippets, PAA, ads, local pack, and metadata like total results.

When to use it

• When you need automated daily or periodic keyword ranking checks.
• To collect People Also Ask and related searches for content gap analysis.
• For competitive intelligence on organic rankings and ad placements.
• When monitoring local pack and city-specific results using geo-targeted proxies.
• To build datasets for SERP feature tracking and SEO reporting.

Best practices

• Always use a real browser automation tool (not plain HTTP clients) to avoid TLS fingerprinting blocks.
• Route traffic through residential proxies and rotate country endpoints for geo-accurate SERPs.
• Throttle requests: use 3–15 second delays depending on rotation strategy to minimize CAPTCHAs.
• Accept cookie consent banners in EU contexts before extracting results.
• Design extraction logic to handle varying SERP layouts and A/B tests (featured snippets, AI overviews).
• Store timestamps, proxy country, and domain used for each scrape to ensure reproducibility.

Example use cases

• Daily rank tracking for target keywords across multiple countries with timestamped history.
• Competitive monitoring to capture competitor organic positions, ad copy, and landing pages.
• Content ideation by scraping People Also Ask and related searches to find unanswered queries.
• Local SEO auditing by scraping local pack results from city-specific proxies.
• Bulk keyword sweeps with rotating IPs to build large-scale SERP datasets for analysis.