evidence-auditor_skill

Overview

This skill audits the evidence supporting each claim in a research artifact and writes concrete gaps and risks into output/MISSING_EVIDENCE.md. It flags missing or weak evidence, prescribes minimal, actionable fixes, and classifies severity to guide author revisions. The guardrail restricts the auditor to identifying gaps only — it does not add new claims or rewrite arguments.

How this skill works

The auditor reads output/CLAIMS.md and, for each claim, locates supporting material in the manuscript or notes that evidence is not locatable. For empirical claims it checks dataset/task definitions, baselines, evaluation protocol, and ablation needs; for conceptual claims it inspects definitions, assumptions, and scope. Results are written claim-by-claim into output/MISSING_EVIDENCE.md with fields: Claim, Evidence present, Gap / concern, Minimal fix, and optional Severity.

When to use it

• During peer review or internal review of a manuscript
• When refining a claims list before submission
• When assessing whether experimental support matches stated conclusions
• When preparing rebuttals or revision plans after reviewer feedback

Best practices

• Run only after output/CLAIMS.md exists and claims are explicit
• Keep evidence pointers specific (section, figure, table, or line numbers) to avoid ‘not locatable’ flags
• Label each gap with the smallest concrete addition that would resolve it (what dataset, metric, baseline, or analysis)
• Avoid proposing new claims or re-arguing the paper — focus on gaps, risks, and next-step validation
• Use Severity to prioritize fixes: major for claims that undermine conclusions, minor for refinements

Example use cases

• Identify that a reported performance claim lacks a clear baseline and request one comparable model and metric
• Find that a dataset description omits preprocessing steps and ask for a reproducible pipeline description
• Detect that a theoretical claim relies on an unstated assumption and require explicit statement and proof sketch
• Note missing ablations for hyperparameters and request sensitivity analyses
• Flag unsupported generalization statements and request held-out or cross-domain evaluation