agent-evaluation_skill

Overview

This skill guides the design and implementation of comprehensive evaluation systems for AI agents. It covers grader types (code-, model-, human-based), benchmarks for different agent classes, an 8-step roadmap for building eval suites, and how to integrate evals into production and CI/CD. Use it to create repeatable, outcome-focused, and maintainable agent evaluations.

How this skill works

The skill explains what to inspect: tasks, trials, transcripts, outcomes, and grader outputs. It prescribes grader selection and rubric design per agent type, test harness and isolation patterns, and metrics to collect (pass rates, variance, resolution rate, etc.). It also describes workflows for running suites in CI, sampling in production, A/B testing, and monitoring eval saturation and maintenance.

When to use it

• Building benchmarks for coding, conversational, research, or computer-use agents
• Designing graders and rubrics for objective scoring
• Implementing CI/CD pipelines with automated eval runs
• Setting up production monitoring and sampling for model quality
• Debugging agent behavior via transcript analysis
• Measuring improvement and detecting eval saturation

Best practices

• Start small: 20–50 representative tasks covering common, edge, and failure cases
• Prefer code-based graders for structured outputs; use model graders for nuance and humans for final validation
• Focus grading on final outcomes rather than intermediate steps
• Isolate eval environments (containers, timeouts, resource limits) to avoid flakiness
• Regularly read transcripts to identify failure modes and tool usage patterns
• Monitor saturation and retire or harden tasks when pass rates are consistently high

Example use cases

• Coding agent: run test suites, lint, build checks, and compute test-passage rate for each trial
• Conversational agent: apply a multi-dimensional rubric (empathy, resolution, efficiency) with model-based grader and aggregated scores
• Research agent: score grounding, coverage, source quality, and factual accuracy with a blend of automated checks and human review
• Computer-use agent: verify UI state, file/db state, screenshots and outcome conditions in an isolated sandbox
• CI integration: run compact eval subsets on PRs and full suites on nightly builds with artifact upload