evaluation_skill

Overview

This skill evaluates LLM outputs with multi-dimensional rubrics, supports handling non-determinism, and implements the LLM-as-judge pattern for repeatable quality checks. It focuses evaluation effort on prompt design and structured test cases so teams can reliably compare prompts, models, and changes. Use it to create automated evaluation pipelines and enforce quality gates before production deployment.

How this skill works

Define a rubric with weighted dimensions (accuracy, completeness, clarity, conciseness, format) and author structured test cases that include expected behavior and optional ground truth. Run multiple inference runs per test to measure mean and variance, then aggregate scores per-dimension and overall. Optionally use a stronger model as an automated judge: feed outputs plus rubric to the judge LLM to produce scored evaluations and written feedback.

When to use it

• Before shipping prompts or model changes to production
• When A/B testing prompt variations or model versions
• During regression testing to detect output degradation
• When building evaluation datasets or automated quality gates
• Any time you need reproducible, multi-dimensional output assessments

Best practices

• Design multi-dimensional rubrics; avoid single overall scores
• Run each prompt 3–5 times and report mean and variance
• Prefer prompt improvements over model hyperparameter tweaks
• Use a stronger model as judge and validate judge consistency
• Require statistical thresholds (e.g., ≥70% win rate) for claims of improvement

Example use cases

• Compare two prompt rewrites for instruction clarity using rubric scores
• Validate that a model update did not regress accuracy on a core test suite
• Automate quality gates in CI: fail build if average score falls below threshold
• Use judge LLM to scale human evaluation for large result sets
• Create labeled failure cases for prompt engineering and remediation