evaluating-llms_skill

Overview

This skill evaluates Large Language Model (LLM) systems using automated metrics, LLM-as-judge patterns, and standardized benchmarks to ensure quality, safety, and readiness for production. It provides a decision framework for selecting evaluation approaches by task, volume, and cost, plus concrete patterns for unit tests, RAG assessment, safety checks, benchmarks, and production monitoring.

How this skill works

The skill inspects model outputs with layered evaluation: fast automated checks for structure and format, LLM-as-judge for nuanced quality scoring, and human review for edge cases. It integrates libraries and benchmarks (RAGAS, lm-eval, scikit-learn, HumanEval) and measures metrics like faithfulness, EM/F1, BLEU/ROUGE/BERTScore, Pass@K, and toxicity scores. Results drive prompt fixes, retrieval tuning, model selection, and CI/CD gating.

When to use it

• Testing and validating individual prompt correctness and output format
• Evaluating and monitoring RAG pipeline faithfulness and relevance
• Measuring hallucinations, bias, or toxicity before deployment
• Comparing models or prompts for A/B testing and production selection
• Running standardized benchmarks (MMLU, HumanEval, HellaSwag) for capability baselining
• Integrating automated LLM quality checks into CI/CD and production monitoring

Best practices

• Use a layered evaluation: automated metrics for all outputs, LLM-as-judge for a sample, humans for critical edge cases
• Choose metrics by task: accuracy/F1 for classification, EM/F1 for QA, Pass@K for code, faithfulness for RAG
• Design clear rubrics and few-shot examples for LLM-as-judge to improve correlation with human judgment
• Monitor faithfulness first in RAG systems; prioritize grounding and citation requirements to reduce hallucinations
• Average multiple LLM-judge runs to reduce variance and watch for judge biases
• Automate checks (JSON schema, regex, keyword presence) to catch format regressions in CI/CD

Example use cases

• Unit evaluation of JSON-formatted extraction and automated pytest assertions
• RAG pipeline audit using RAGAS metrics to measure faithfulness and context relevance
• LLM-as-judge scoring of summary quality with a 1–5 rubric for product support answers
• Benchmarking GPT-4 vs Claude on MMLU and HumanEval to guide model selection
• Online monitoring that samples responses for LLM-judge scoring and routes low-confidence outputs to human review