raveeshbhalla/dspy-gepa-logger

Overview

This skill guides users to optimize LLM prompts using a custom implementation of the GEPA (GEnetic PAreto) algorithm. I walk you through validating prerequisites, preparing your data and grader, generating DSPy scripts, running an observable GEPA run, and handing off an optimized prompt. The focus is reproducible runs, meaningful textual feedback, and Pareto-driven prompt selection for reliable performance.

How this skill works

I inspect your dataset, help map inputs/expected outputs, and produce DSPy objects and a grader that returns both a numeric score and textual feedback. The custom GEPA implementation uses textual feedback plus Pareto frontiers to propose and evolve candidate prompts. I also scaffold demo scripts, start a local web dashboard for live observability, and validate the optimized prompt with smoke tests.

When to use it

• When you have a dataset and want systematic prompt improvement beyond manual tweaking.
• When outputs require nuanced judgment best captured by LLM feedback, not only scalar rewards.
• When you want reproducible optimization with visibility into failures and log traces.
• When you need a robust prompt that performs consistently across many cases, not just best average score.
• When you plan to deploy a tuned prompt into a production Task LM and want a clear handoff.

Best practices

• Provide a representative validation set and map inputs/outputs clearly before running optimization.
• Use a stronger Reflective/Judge LM than your production Task LM for high-quality textual feedback.
• Start with auto=light budget to validate wiring, then increase to medium/heavy for final runs.
• Keep .env.local API keys private and confirm you manually added them before running tests.
• Monitor the dashboard at http://localhost:3000 while the optimizer runs and keep the server terminal open.

Example use cases

• Improving an instruction prompt for classification across a messy CSV with category labels.
• Optimizing a prompt that generates structured JSON outputs from free-text inputs where judges provide corrective feedback.
• Tuning chain-of-thought prompts for a reasoning task and selecting prompts that dominate on most validation cases.
• Converting a rule-based rubric into an LLM judge and iterating prompts until feedback quality stabilizes.
• Running reproducible experiments for prompt transfer from a small dev set to a larger production dataset.