apify_skill

This skill helps you automate data extraction from websites using Apify, running pre-built actors and managing datasets to save time.

Shell

39

GitHub Stars

1

Bundled Files

2 months ago

Catalog Refreshed

4 months ago

First Indexed

Readme & install

Copy the install command, review bundled files from the catalogue, and read any extended description pulled from the listing source.

Installation

Preview and clipboard use veilstrat where the catalogue uses aiagentskills.

npx veilstrat add skill vm0-ai/vm0-skills --skill apify

SKILL.md10.6 KB

Overview

This skill integrates with Apify, a web scraping and automation platform that runs pre-built Actors or custom scrapers. It enables running scrapers, polling run status, fetching datasets, and managing runs via simple shell/curl workflows. The skill targets automated extraction of structured data from websites at scale.

How this skill works

It uses Apify HTTP API endpoints to start Actor runs (async or sync), poll run status, retrieve dataset items, list Actors and runs, and abort jobs. Workflows rely on an APIFY_API_TOKEN environment variable and standard tools like curl, jq and bash for capture/poll loops. Common Actors (Google, Amazon, Instagram, website crawler) are invoked by posting JSON input to the appropriate /acts endpoint.

When to use it

You need structured data from websites (products, search results, profiles).
You want to run ready-made scrapers without writing a full scraper.
You must automate repeated web tasks at scale and persist outputs.
You need to orchestrate long-running crawls asynchronously and poll status.
You want to fetch results directly with sync endpoints for short tasks (<5 min).

Best practices

Set APIFY_API_TOKEN as an environment variable and use bash -c when piping to avoid variable clearing.
Choose run-sync-get-dataset-items for quick jobs (<5 minutes) and async runs for longer crawls.
Capture runId and defaultDatasetId from the start response for polling and fetching results.
Set explicit timeout and memory options for sync runs to avoid unexpected truncation or high costs.
Use pagination (limit/offset) for large datasets and respect rate limits to avoid throttling.

Example use cases

Run a Google Search Actor to collect SERP links and snippets for keywords.
Crawl a documentation site to extract page titles and URLs using the website-content-crawler.
Scrape Amazon product details (price, title, reviews) with the Amazon crawler Actor.
Collect recent Instagram posts or a Twitter feed using the corresponding Actor.
Automate recurring crawls and store results in datasets for downstream ETL and analytics.

FAQ

Create an Apify account and set APIFY_API_TOKEN in your environment, then include it in Authorization: Bearer ${APIFY_API_TOKEN} headers.

When should I use sync vs async runs?

Use sync endpoints for quick tasks under about 5 minutes to get results directly; use async runs for longer or resource-heavy crawls and poll run status to detect completion.

How do I get results after starting an async run?

Capture data.id (runId) and data.defaultDatasetId from the start response. Poll the actor-run endpoint until status is SUCCEEDED, then fetch items from /datasets/{datasetId}/items.