eval-harness_skill

Overview

This skill provides a comprehensive evaluation framework for systematic testing, measurement, and quality assurance of AI-assisted implementations. It supports capability evaluations, regression testing, multiple grader types (code, model, human), and standardized metrics to track progress and prevent regressions. The harness is designed to embed evaluative criteria into development workflows so tests act as executable specifications.

How this skill works

The harness defines structured evals (capability and regression) that pair explicit inputs with expected outputs and grader configurations. Graders run programmatically or via model/human review: code graders perform exact/regex/function tests, model graders score semantic outputs with rubrics, and human graders capture subjective judgments. Results are recorded with metadata, scoring, and tolerances to enable automated gating, historical baselines, and trend analysis.

When to use it

• Define success criteria before or alongside feature implementation (evaluation-driven development).
• Validate new capabilities such as code generation, information extraction, or API behavior.
• Prevent regressions after refactors, dependency updates, or model/prompt changes.
• Automate CI checks and quality gates for PRs or release pipelines.
• Measure improvements over time and compare model/prompt variants.

Best practices

• Write clear, minimal expected outputs and prefer automated graders for deterministic checks.
• Capture a known-good baseline for every protected behavior and store environment details.
• Use model-based graders with explicit rubrics and human review for subjective or high-risk evaluations.
• Set sensible timeouts, retry policies, and weights to prioritize critical evals.
• Treat failed evals as actionable items: reproduce, annotate root cause, and add or update tests to prevent recurrence.

Example use cases

• Capability eval: verify a generated Python function passes unit-style test cases using a function_output grader.
• Regression eval: assert API response structure remains compatible while allowing tolerated field changes via structural comparison.
• Semantic eval: use a model grader with a rubric to score document information extraction for completeness and accuracy.
• CI integration: run a suite of evals on each PR to block merges that introduce critical regressions.
• Benchmarking: compare prompt or model variants by running identical eval sets and recording numeric scores over time.