evaluation_skill

Overview

This skill provides a practical framework for evaluating agent systems across multiple dimensions. It focuses on outcome-centered rubrics, realistic test sets, and continuous evaluation pipelines to catch regressions and measure improvements. The guidance balances automated LLM-as-judge methods with human review for edge cases.

How this skill works

The skill defines multi-dimensional rubrics (accuracy, completeness, citation accuracy, source quality, tool efficiency) and converts them to weighted numeric scores. It supports complexity-stratified test sets, token-budgeted runs, and automated LLM-based judgments while recommending human sampling for subtle failures. Results are tracked over time to detect regressions and validate context engineering choices.

When to use it

• When you need a systematic test framework for agent performance
• Before deploying agent changes to catch regressions
• To compare agent configurations, models, or context strategies
• When building quality gates and automated evaluation pipelines
• To measure production quality by sampling real interactions

Best practices

• Design multi-dimensional rubrics; avoid single-metric decisions
• Evaluate outcomes, not specific execution paths or steps
• Stratify test sets by complexity and include edge cases
• Run evaluations under realistic token budgets and context sizes
• Combine LLM-as-judge for scale with human review for edge cases

Example use cases

• Compare two agent architectures by running the same test set and comparing weighted scores
• Build a CI pipeline that runs evaluation tests on every agent version and flags regressions
• Measure the impact of context-window reductions with degradation tests to find safe limits
• Use LLM-as-judge prompts to score thousands of runs, then human-review low-confidence failures
• Create pass/fail quality gates that enforce minimum weighted scores before deployment