agentic-eval_skill

Overview

This skill provides patterns and techniques for evaluating and improving AI agent outputs through iterative critique and refinement. It focuses on practical evaluation loops—self-reflection, separate evaluator-optimizer pipelines, and test-driven code refinement—to raise quality for code, reports, and analysis. The guidance emphasizes structured outputs, clear criteria, and iteration controls for reliable, automatable improvement.

How this skill works

The skill outlines three core patterns: a Basic Reflection loop where the agent self-critiques and refines; an Evaluator-Optimizer pattern that separates generation, scoring, and optimization; and a Code-Specific Reflection loop using tests to drive fixes. Each pattern uses structured outputs (JSON or numeric scores) from the evaluator to decide whether to accept, refine, or stop. Iteration limits, convergence checks, and logging are recommended to avoid infinite loops.

When to use it

• When generation quality is critical (production code, compliance documents, data analysis).
• When you can define explicit success criteria or a rubric for evaluation.
• When you want automated iterative improvement rather than single-shot outputs.
• When building LLM-as-judge or rubric-based scoring pipelines.
• When implementing test-driven refinement for generated code.

Best practices

• Define clear, measurable evaluation criteria or a weighted rubric before generating.
• Return evaluation results in structured formats (JSON, numeric scores) for reliable parsing.
• Limit iterations (typically 3–5) and add convergence checks to stop non-improving loops.
• Log full iteration history for debugging and performance analysis.
• Separate responsibilities: generation, evaluation, and optimization for clearer workflows.

Example use cases

• Generate production-ready code with pytest-driven loops that run tests and patch failures.
• Produce compliance reports evaluated against a rubric of accuracy, clarity, and completeness.
• Implement an LLM-as-judge system to rank multiple candidate responses for a support bot.
• Build an optimizer that reruns refinement until an overall score threshold is met.
• Integrate self-critique for drafts of technical documentation to improve clarity and correctness.