agent-scorecard_skill

Overview

This skill provides a configurable output-quality evaluation framework for AI agents. It lets you define measurable quality dimensions, run pattern-based automated checks or manual rubrics, and track quality trends over time. Everything runs locally with no external API calls and no LLM-as-judge.

How this skill works

You define dimensions, rubrics, weights, and pass/fail thresholds in a configuration file. The engine runs automated pattern checks (format compliance, filler words, required sections, style consistency, length) or guides a human through manual scoring, and can blend both. Results are aggregated per your chosen method and appended to a local history for trend analysis, comparisons, and report generation.

When to use it

• Validate whether prompt or config changes improved agent outputs
• Automate regression checks after adding tools or code changes
• Compare output quality across models or agent versions
• Establish team-wide rubrics and pass/fail thresholds
• Monitor long-term output quality and detect degradation

Best practices

• Start with a small set of clear dimensions (accuracy, completeness, format) and expand as needed
• Set realistic thresholds and weights reflecting business priorities
• Use blended mode to combine fast automated checks with targeted human review
• Store history centrally and run periodic trend analyses to catch slow regressions
• Treat config files as trusted: review any external config before loading due to code execution risk

Example use cases

• Prompt engineering experiments: measure objective before/after impact
• Model comparison: run identical tasks across models to rank quality
• Agent regression testing: run automated checks in CI to catch regressions
• Operational monitoring: daily or weekly scoring to detect slow drift
• Team QA: shared rubrics for consistent manual reviews across reviewers