prompt-testing_skill

Overview

This skill runs A/B tests and measures prompt performance using defined metrics and a repeatable protocol. It helps teams compare prompt variants, quantify trade-offs (quality, efficiency, robustness), and produce actionable recommendations. The focus is on reproducible results and clear decision criteria.

How this skill works

It defines objectives and success criteria, prepares variants and a labeled test dataset, executes runs to collect responses and telemetry, then computes metrics like accuracy, compliance, tokens, latency, and edge-case handling. Results are aggregated into a structured report that highlights deltas, problematic cases, statistical confidence, and a recommended action. Commands support creating tests, running A/B comparisons, viewing results, and comparing past tests.

When to use it

• Selecting between two prompt designs before deployment
• Measuring impact of prompt edits on quality and cost
• Validating robustness against edge cases and jailbreak attempts
• Tracking regression across prompt versions
• Benchmarking prompts across models or configurations

Best practices

• Define clear test objectives and hypothesis before running tests
• Use a minimum of 20 diverse cases with 15–20% edge cases
• Run multiple repeats to measure consistency and variance
• Version and document each prompt variant and dataset used
• Report both absolute metrics and per-case breakdowns for transparency

Example use cases

• Compare a concise instruction prompt vs a detailed walkthrough to see accuracy vs tokens trade-off
• Evaluate a defensive prompt change for jailbreak resistance on adversarial inputs
• Measure latency and token cost differences when moving prompts between models
• Validate that a candidate prompt does not regress on existing edge-case behaviors
• Produce an A/B test report to justify adopting a new prompt in production