advanced-evaluation_skill

Overview

This skill provides production-grade patterns for using LLMs as judges to evaluate model outputs, generate rubrics, and build reliable automated evaluation pipelines. It distills practical protocols for direct scoring, pairwise comparison, bias mitigation, and metric selection. Use this when you need repeatable, debuggable judgments and calibrated confidence for large-scale or high-stakes evaluation. The guidance emphasizes evidence-first scoring, position-swap checks, and domain-adapted rubrics.

How this skill works

The skill codifies two primary evaluation modes: direct scoring (single-response ratings on calibrated scales) and pairwise comparison (head-to-head preference judgments with position-swap mitigation). It defines rubric components, prompt structures that require justification before scores, and a bias mitigation layer (position, length, verbosity, self-enhancement, authority). The pipeline combines criteria loading, primary scoring, bias mitigation, and confidence calibration to produce scores, justifications, and actionable improvements.

When to use it

• Building automated evaluation pipelines for LLM outputs or A/B tests
• Comparing multiple model responses to select the best or most suitable
• Designing clear, domain-adapted rubrics for human or automated judges
• Mitigating evaluation biases like position or length bias
• Scaling evaluation with panels of LLMs, hierarchical screening, or human-in-the-loop

Best practices

• Require evidence-based justification before returning any numeric score
• Use pairwise comparison for subjective preferences and direct scoring for objective facts
• Swap positions and perform consistency checks to eliminate position bias
• Keep one measurable aspect per criterion; avoid overloaded criteria
• Calibrate confidence from position consistency and strength of evidence
• Build domain-specific level descriptions, examples, and edge-case guidance in rubrics

Example use cases

• Direct scoring factual accuracy and instruction following with 1–5 scales and justifications
• Pairwise comparisons for tone, style, or persuasiveness with position-swap protocol
• Generating a code-readability rubric with level descriptions, characteristics, and edge cases
• High-volume evaluation using a cheap screening model plus an expensive adjudicator for edge cases
• Panel-of-LLMs aggregation to reduce individual judge bias for critical decisions