building-screening-rubrics_skill

Overview

This skill helps teams build, validate, and iterate paper-screening rubrics for literature reviews and large-scale screening. It emphasizes collaborative brainstorming, test-driven validation on a small ground-truth set, and automated bulk screening once the rubric meets accuracy targets. The workflow produces versioned JSON artifacts for reproducibility and re-screening.

How this skill works

You define relevance criteria through guided questions (keywords, data types, paper types, edge cases) and encode them into a scoring rubric (weights, thresholds, special rules). Then you assemble a 5–10 paper test set, collect user judgments, apply the rubric to compute predicted scores, and measure accuracy. Iterate on rules until the rubric meets a target (typically ≥80%), then run bulk screening and cache abstracts for future re-scoring.

When to use it

• Starting a literature search that will screen 50+ papers
• Current rubric is producing many false positives or false negatives
• Need to define relevance before automating screening
• Re-screen cached results after updating inclusion criteria
• Preparing helper scripts that depend on reproducible scoring logic

Best practices

• Brainstorm edge cases and exclusion terms up front and save them in screening-criteria.json
• Validate on 5–10 labeled papers and aim for ≥80% accuracy before bulk screening
• Cache all fetched abstracts to avoid repeated API calls and enable fast re-screening
• Prefer additive, explainable scoring with a small number of special rules
• Document rubric changes and reasons in a changelog with versioning

Example use cases

• Design a rubric to find papers with IC50 measurements and public datasets in medicinal chemistry
• Iteratively refine rules to exclude reviews and prioritize primary research
• Build a test set from PubMed, validate scoring, then bulk-screen 200 search results
• Update rubric after offline review and re-score cached abstracts to produce a change report
• Parameterize helper scripts with screening-criteria.json for reproducible automation