setup-scheduled-scraper_skill

Overview

This skill sets up a local Playwright + Next.js scraper project that runs on a schedule and stores results as JSON. It scaffolds a TypeScript scraper, launchd scheduling for macOS, logging scripts, and an optional Next.js App Router + Tailwind + Shadcn web viewer. The output is ready-to-run with sensible defaults and helpers to customize schedule times and output locations.

How this skill works

The skill scaffolds a project with a Playwright entry (scraper.ts) and a test spec that invokes it, writing structured records to results.json for scheduled runs and results-local.json for manual runs. It includes shell scripts to wrap Playwright runs, plist files for LaunchAgent/LaunchDaemon scheduling, optional pmset wake helpers, and a read-only Next.js viewer wired to the JSON output. Environment variables (e.g., SCRAPE_RESULTS_PATH) and metadata files (scraper-metadata.json) let you control output locations and track run details.

When to use it

• You need a reproducible local scraper that runs on a schedule (macOS).
• You want Playwright-based scraping with TypeScript and test-driven invocation.
• You want scheduled JSON output and simple run metadata for downstream processing.
• You want an optional read-only Next.js viewer for browsing results locally or on Vercel.
• You need launchd-compatible automation with wake-from-sleep support.

Best practices

• Keep project-specific labels, filenames, and log paths unique across machines to avoid plist collisions.
• Use results.json for scheduled runs and results-local.json for manual troubleshooting; support SCRAPE_RESULTS_PATH to override paths.
• Test scraping locally with npm run scrape before enabling the LaunchAgent; verify logs with tail and launchctl list.
• Limit the LaunchAgent to two default daily times; ask before expanding to more intervals and use update-schedule.sh to edit StartCalendarInterval.
• Keep the Next.js viewer read-only and compute derived views from filtered subsets to avoid heavy runtime processing.

Example use cases

• Twice-daily scrape of public scoreboard pages; results.json drives a local dashboard and notifications.
• Periodic extraction of product prices for comparison; scheduled runs append fresh snapshots and scraper-metadata.json stores timestamps and counts.
• Nightly capture of event listings with optional pmset wake scheduling to ensure machine wakes before scrape.
• Local development setup where manual runs write results-local.json for debugging while scheduled runs write results.json for the viewer.