deepeval_skill

Overview

This skill helps you evaluate LLM applications using DeepEval, a pytest-based evaluation framework focused on RAG, conversational AI, agents, safety, and custom metrics. It streamlines metric-driven testing, component tracing, and CI/CD integration to validate model behavior and guardrails. Use it to run reproducible tests, measure faithfulness and relevancy, and automate evaluations in pipelines.

How this skill works

The skill uses DeepEval test case types (single-turn, conversational, multimodal) and 50+ built-in metrics to score outputs against expected behavior. You can instrument components with @observe for span-level tracing, run asynchronous or cached evaluations, and configure providers per-metric or globally. Results integrate with pytest and can be automated in CI/CD workflows.

When to use it

• Validating RAG pipelines (retrieval + generation)
• Testing multi-turn conversational assistants and role adherence
• Measuring agent tool usage and task completion
• Running safety gates for toxicity, bias, or PII leakage
• Automating model checks in CI/CD before deployment

Best practices

• Start with 2–3 core metrics and add 1–2 custom metrics only as needed
• Use component-level tests with @observe before end-to-end runs
• Prefer Anthropic Sonnet/Haiku models when available for evaluation
• Keep thresholds moderate in development (0.5–0.6) and tighten for production (0.7–0.8)
• Use async and cache configs to balance throughput and rate limits; reduce concurrency if throttled

Example use cases

• RAG evaluation: measure contextual precision, recall, relevancy, answer relevancy and faithfulness on retrieved chunks
• Conversational evaluation: validate role adherence, knowledge retention, turn relevancy across multi-turn dialogues
• Agent evaluation: confirm correct tool selection, argument correctness, and task completion for tool-using agents
• Safety gate: run toxicity, bias, and PII checks and return pass/fail with failure reasons
• CI automation: run deepeval test run in GitHub Actions to fail PRs on regressions