pdf-text-extractor_skill

Overview

This skill downloads available PDFs and extracts plain text to provide full-text evidence for paper notes and claims. It writes a record per attempt to papers/fulltext_index.jsonl and saves extracted text under papers/fulltext/*.txt with cached PDFs in papers/pdfs/. The design is conservative: it skips downloads in abstract mode and avoids overwriting existing extractions unless explicitly requested.

How this skill works

The skill reads papers/core_set.csv (and optionally outline/mapping.tsv) to prioritize papers. In fulltext mode it resolves pdf_url (or derives from arxiv_id/url), downloads missing PDFs to papers/pdfs/<paper_id>.pdf, extracts a prefix of text to papers/fulltext/<paper_id>.txt, and appends status and stats to papers/fulltext_index.jsonl. It supports a local-pdfs-only mode where no network downloads occur and produces machine- and human-readable to-do lists for missing PDFs.

When to use it

• You set evidence_mode: fulltext in queries.md and need stronger, full-text evidence beyond abstracts.
• You want cached PDFs and extracted plain text for downstream claim verification or quoting.
• You can allow network downloads or have provided local PDFs in papers/pdfs/.
• You need a reproducible index of which papers were attempted, skipped, or failed.
• You want lightweight extraction (configurable max pages/min chars) to limit cost/time.

Best practices

• Default to evidence_mode: abstract for broad surveys and enable fulltext selectively to control cost.
• Populate pdf_url or arxiv_id in papers/core_set.csv to improve download success rates.
• Use --local-pdfs-only when network access is restricted or when you pre-populate papers/pdfs/.
• Do not delete existing papers/fulltext/*.txt unless you intend to re-run extraction for that paper.
• Tune max-papers, max-pages, and min-chars to match your compute/budget constraints.

Example use cases

• Run targeted fulltext extraction for 20 high-priority papers to support evidence-heavy claims in a literature review.
• Provide full-text snippets for a claim verification pipeline that needs passages longer than abstracts.
• Create a cached corpus of PDFs and extracted text for offline text-mining experiments.
• Fallback to local-pdfs-only mode in restricted environments and generate missing_pdfs lists for manual download.