advanced-evaluation_skill

Overview

This skill teaches production-grade LLM-as-a-Judge evaluation techniques including direct scoring, pairwise comparison, rubric generation, and bias mitigation. It focuses on choosing the right evaluation approach, designing calibrated prompts and rubrics, and building pipelines that produce reliable, auditable judgments. Use it to design automated evaluation systems, compare model outputs, or set consistent quality standards.

How this skill works

The skill describes concrete implementations: direct scoring with evidence-first justification and calibrated scales, pairwise comparison with position-swap consistency checks, and rubric generation with level descriptions and edge-case guidance. It also maps metrics to task types, prescribes bias-mitigation protocols (position, length, verbosity, self-enhancement), and shows how to aggregate judges, calibrate confidence, and integrate human review for low-confidence cases.

When to use it

• Building automated evaluation pipelines for model outputs
• Comparing multiple model responses to select the best one
• Creating or calibrating rubrics for human or automated evaluation
• Designing A/B tests for prompt or model changes
• Debugging inconsistent evaluation outcomes

Best practices

• Require justification before any numeric score to improve reliability and auditability
• Choose pairwise comparison for subjective preferences and direct scoring for objective criteria
• Mitigate position bias by swapping response order and applying consistency rules
• Keep criteria atomic: one measurable aspect per criterion to reduce variance
• Use panels of diverse LLM judges or hierarchical screening to scale reliably

Example use cases

• Direct scoring factual accuracy for QA systems with a 1–5 calibrated rubric
• Pairwise comparison to choose the preferred marketing copy based on tone and persuasiveness
• Generate domain-adapted rubrics (code readability, medical accuracy) for consistent team evaluation
• Deploy a screening model for bulk evaluation and route low-confidence cases to a stronger model or humans
• Aggregate votes from a panel of models to reduce single-model bias in high-stakes decisions