omidzamani/dspy-skills

Overview

This skill provides a lightweight mini-batch Bayesian optimizer (SIMBA) for DSPy programs. It focuses on optimizing prompts and agent pipelines using custom feedback signals rather than only accuracy. SIMBA is designed as a budget-friendly alternative to heavier methods like GEPA while supporting nuanced metric design.

How this skill works

SIMBA runs iterative mini-batch optimization: it samples batches, evaluates candidates with a user-provided metric, and proposes prompt edits or demonstrations that improve scores. It leverages statistical analysis of feedback and output variability to target hard examples and generate self-reflective rules or demonstrations. The optimizer supports both simple scalar metrics and rich dspy.Prediction(score, feedback) signals for finer-grained guidance.

When to use it

• You need a lighter-weight alternative to GEPA for agent/program optimization
• Your evaluation uses custom or nuanced feedback signals (not just binary accuracy)
• Budget constraints require fewer evaluation calls and smaller iteration costs
• Agentic systems with rich failure traces where targeted fixes help
• Programs where few-shot example counts are flexible and not critical

Best practices

• Provide a metric that returns dspy.Prediction(score, feedback) to unlock richer adaptations
• Tune bsize and max_steps to balance variance and compute (bsize default 32, max_steps default 8)
• Use lower sampling temperatures (0.1–0.3) to exploit strong candidates and higher to explore
• Start with smaller num_candidates and max_demos, then scale up after observing behavior
• Validate optimized programs on a held-out dev set to detect overfitting to noisy feedback

Example use cases

• Optimizing a QA pipeline where partial matches or verbosity matter and you need nuanced scoring
• Tuning a ReAct research agent with tool calls and multi-step traces using custom feedback
• Improving prompt templates for budget-limited production systems where fewer evals are allowed
• Iteratively adding demonstrations to handle high-variance examples identified by mini-batches
• Using non-accuracy metrics (e.g., coverage, safety flags, or term overlap) to guide optimization