rag-eval_skill

Overview

This skill evaluates the output quality of a RAG pipeline using Ragas metrics: faithfulness, answer relevancy, and context precision. It requires an existing RAG system integrated with OpenClaw (vector DB + retrieval) and at least one LLM API key for the internal judge. Use it to test changes, detect hallucinations, and run batch regression tests on golden datasets. It does not provide retrieval or embedding functionality itself.

How this skill works

The skill accepts a question, the LLM answer, and the list of retrieved context chunks that produced the answer. It calls the configured LLM judge (OpenAI, Anthropic, or a local Ollama LLM) to score faithfulness, relevancy, and context precision, then returns numeric scores, an overall score, and a verdict. You can run single evaluations, request sentence-level explanations for low faithfulness, or run batch evaluations over JSONL datasets to produce date-stamped reports.

When to use it

• After changing retrieval, chunking, or embedding configuration to measure impact
• Testing for hallucinations in retrieved-context answers
• Running nightly or CI batch regression tests against a golden dataset
• Comparing RAG performance before and after model/embedding upgrades

Best practices

• Ensure OpenClaw RAG pipeline is already integrated and returns the exact context chunks used to generate answers
• Provide full, representative context chunks — truncated or altered chunks will skew faithfulness scores
• Use the --explain flag when faithfulness < 0.80 to get sentence-level unsupported claims
• Run batch evaluations on a stable golden dataset to track regressions over time
• Store results (JSONL) with timestamps for trend analysis and automated alerts

Example use cases

• Validate a config change that adjusts chunk size and measure context precision delta
• Detect if a new embedding model increases hallucination rates by comparing faithfulness scores
• Run a nightly batch_eval on a customer support Q&A golden set to catch regressions
• Audit a retrieval tweak that increased relevancy but lowered context precision to find the trade-off