dataforseo-onpage-api_skill

Overview

This skill connects to the DataForSEO OnPage API to crawl and audit sites for technical SEO, site health, and Lighthouse performance metrics. It runs task-based crawls or live checks, normalizes results, and returns both a prioritized summary and raw payloads for debugging. Use it to detect broken links, redirect chains, non-indexable pages, duplicate tags, and performance regressions.

How this skill works

I create crawls via the OnPage task_post endpoint or run single live requests when appropriate. I poll tasks_ready (or accept webhooks) until tasks complete, then fetch summary, pages, resources, and Lighthouse results. Responses are validated (status_code checks), normalized into a concise issue list with severity and URL context, and the raw API payload is made available for troubleshooting. For in-progress work I can force_stop crawls on demand.

When to use it

• Run a full site crawl after a deployment to catch regressions (redirects, noindex, broken links).
• Audit a site for duplicate titles, meta tags, or canonical issues across many pages.
• Generate Lighthouse performance reports at scale for a set of URLs and detect regressions.
• Create an automated QA gate for SEO before release or scheduled site-health monitoring.
• Prioritize remediation work by combining technical issues with Lighthouse performance metrics.

Best practices

• Use task-based mode for large sites and scheduled monitoring; use live mode for single-URL checks.
• Provide HTTP Basic credentials (login:password) and verify access to the target domain before running large crawls.
• Limit crawl depth and page limits during initial runs to avoid unnecessary load.
• Store tasks[].id and poll tasks_ready or configure postback_url to avoid long synchronous waits.
• Always surface the raw API payload with the normalized summary for developer troubleshooting.

Example use cases

• Crawl an entire domain to list broken links, 4xx/5xx responses, and redirect chains.
• Run Lighthouse on 50 high-priority pages and export the slowest resources and opportunity suggestions.
• Detect non-indexable pages (noindex, disallowed by robots) after a migration and produce a prioritized fix list.
• Scan for duplicate titles/meta across product pages and output CSV-ready exports for content teams.
• Integrate Lighthouse metrics into an engineering backlog to track performance improvements over time.