bigcode-evaluation-harness_skill

Overview

This skill evaluates code generation models using the BigCode Evaluation Harness across 15+ benchmarks (HumanEval, MBPP, MultiPL-E and more). It produces standardized pass@k metrics and saves generation outputs for analysis. Use it to benchmark models, compare multi-language performance, and generate reproducible results for leaderboards and research.

How this skill works

The harness runs model generation over selected task suites, optionally executes generated code in a controlled environment, and computes pass@k estimates from multiple samples per problem. It supports generation-only workflows, Docker-based safe execution for multi-language benchmarks, instruction-tuned prompts, and configurable sampling/quantization settings. Outputs are JSON metric files plus saved generations for further analysis.

When to use it

• Benchmarking code-generation models for research or model selection
• Measuring functional correctness with pass@k on Python and 18 other languages
• Comparing models across HumanEval, MBPP, MultiPL-E, APPS, DS-1000, etc.
• Testing instruction-tuned or chat-style code models with special tokens
• Running multi-language evaluations safely using generation-only + Docker

Best practices

• Use n_samples >= 200 for stable pass@k estimates, especially pass@1/10/100
• Set allow_code_execution only in controlled environments; use Docker for multi-language runs
• Keep consistent temperature and max_length_generation across model comparisons
• Save generations to reproduce results and separate generation from execution when needed
• Use 4-bit/8-bit loading to reduce memory footprint and avoid OOM on large models

Example use cases

• Evaluate starcoder2-7b on HumanEval with 200 samples to report pass@1/10/100
• Generate solutions for MultiPL-E on host, then evaluate executions inside the provided Docker image
• Compare three models (StarCoder, CodeLlama, DeepSeek) across HumanEval and MBPP and export a results table
• Test instruction-tuned models with instruct-humaneval and custom instruction tokens
• Run APPS or DS-1000 for large-scale benchmarking of competition or data-science tasks