evaluating-paper-relevance_skill

Overview

This skill performs two-stage paper screening: a fast abstract scoring pass followed by targeted deep dives for papers that pass a relevance threshold. The goal is precision over breadth—find papers that actually contain the data or methods requested, then extract and document those findings. It includes mandatory progress reporting and structured tracking of results.

How this skill works

First, it scores each abstract 0–10 using keyword match, data-type match, and specificity. Papers scoring ≥7 advance to a deep dive where full text is fetched (PubMed Central, DOI resolution, Unpaywall), ChEMBL is checked for medicinal-chemistry SAR, and methods/results/tables/supplementary files are searched. Findings are recorded in a structured summary and every paper is logged to prevent re-review.

When to use it

• You have a search result list and must prioritize which papers to inspect.
• You need specific measurements, protocols, datasets, or code from the literature.
• Screening a small to large corpus (single-review or batch workflows).
• When you must document progress for collaborators or maintain audit trails.
• When medicinal chemistry, genomics, ecology, computational, or clinical data is required.

Best practices

• Always report progress for every paper—never screen silently; show abstract score and decision.
• Use the 0–10 rubric: keywords (0–3), data-type (0–4), specificity (0–3); skip <5, hold 5–6, deep-dive ≥7.
• If paywalled, always try Unpaywall before giving up; ask user for an email if needed for API calls.
• For medicinal chemistry, check ChEMBL before parsing PDFs to leverage curated SAR data.
• Log every paper into papers-reviewed.json and summarize relevant finds in SUMMARY.md to avoid duplication.

Example use cases

• Quickly triage 30 search hits to find papers with IC50 or SAR tables for a drug project.
• Screen genomics hits to locate RNA-seq datasets and GEO accession numbers for meta-analysis.
• Locate computational-method papers with code repositories and benchmark tables for replication.
• Process a large literature sweep with helper scripts that batch abstract fetch, scoring, and resumable deep dives.
• Identify ecology studies reporting population counts and measurement protocols for a systematic review.