dspy-evaluation-suite_skill

Overview

This skill provides a systematic evaluation suite for DSPy programs, letting you run parallelized tests with built-in or custom metrics and produce per-example results and aggregate scores. It is designed to measure correctness, quality, and other dimensions of program outputs so you can compare variants, establish baselines, and validate readiness for production. The suite supports exact-match, LLM-based SemanticF1, and user-defined multi-factor metrics.

How this skill works

You create an Evaluate instance with a devset, a metric function, and thread settings, then call it with a DSPy Module to run the evaluation. The evaluator runs each example (in parallel when configured), computes a score per example using the provided metric, and returns an overall average score plus per-example tuples (example, prediction, score). Built-in metrics include answer_exact_match and SemanticF1; you can also pass metrics that return a scalar or (score, feedback) pairs for optimizer integrations.

When to use it

• Measuring performance before and after prompt or model optimization
• Comparing multiple DSPy program variants to select a best candidate
• Establishing baseline scores for tracking regressions
• Validating production readiness with a held-out test set
• Running automated evaluations as part of an optimization loop

Best practices

• Keep a held-out test/dev set and never tune on evaluation data
• Combine multiple metrics (correctness, conciseness, reasoning quality)
• Use sufficient sample sizes (100+ examples) for statistical confidence
• Monitor cost when using LLM-based metrics like SemanticF1
• Run evaluations in parallel but watch for API rate limits

Example use cases

• Run answer_exact_match across a 1k-example devset to get a baseline accuracy
• Compare baseline, optimized, and fine-tuned program variants and rank by score
• Create a multi-factor metric that weights correctness, brevity, and reasoning
• Use GEPA-compatible metrics to feed back score+feedback into an optimizer
• Export JSON reports summarizing score, counts, and evaluation config for audits