nemo-evaluator_skill

Overview

This skill evaluates large language models across 100+ benchmarks from 18+ harnesses using a container-first, reproducible platform. It supports multi-backend execution (local Docker, Slurm HPC, cloud) and exports results to MLflow, W&B or local formats for enterprise-grade benchmarking. Use it to run standard academic tasks, safety checks, vision-language tests, and large-scale comparisons across models.

How this skill works

The evaluator packages benchmarks and execution logic into containers and runs them against any OpenAI-compatible endpoint or self-hosted API (vLLM, TRT-LLM, NIMs). It orchestrates tasks, manages environment variables, and collects artifacts and metrics per invocation. Multi-backend support lets you run on local Docker, Slurm clusters, or cloud services while preserving reproducibility and exportable results.

When to use it

• You need a single platform covering 100+ benchmarks (MMLU, GSM8K, HumanEval, safety, VLM).
• Benchmark models on Slurm or other HPC with reproducible containerized jobs.
• Compare multiple models on identical tasks and export results to MLflow or W&B.
• Run safety assessments and vision-language evaluations alongside standard NLP benchmarks.
• Integrate reproducible evaluations into CI or research pipelines.

Best practices

• Containerize deployments and ensure NGC Docker credentials are configured for enterprise images.
• Provide required env vars per task (HF_TOKEN, JUDGE_API_KEY) in the evaluation config to avoid runtime failures.
• Start with limit_samples or lower parallelism when testing configs; increase parallelism for full runs.
• Use deterministic settings (temperature=0.0, fixed seeds, num_fewshot matching papers) when comparing models.
• Export runs to a common store (MLflow/W&B/local JSON) to simplify side-by-side comparison.

Example use cases

• Run MMLU, GSM8K, Humaneval on a hosted OpenAI-compatible endpoint to establish baseline scores.
• Launch a Slurm job to evaluate Llama-3.1 across 1k+ samples using 8 GPUs and export metrics to MLflow.
• Compare two models by running a base config twice with different target.api_endpoint overrides and export both results to W&B.
• Execute safety and VLM harnesses to validate content moderation and multimodal capabilities before deployment.
• Quick smoke test using limit_samples=10 and local Docker to validate config and env vars.