bigcode-project/bigcode-evaluation-harness

Health v2 · maintenance only

Adoption not part of the score

1058 stars · 259 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A framework for evaluating autoregressive code generation language models on benchmarks like HumanEval, MBPP, MultiPL-E, and HumanEvalPack. It supports any Hugging Face Hub model, multi-GPU generation with accelerate, and sandboxed Docker execution for reproducible evaluation.

Use cases

• evaluate a code generation model on HumanEval
• benchmark an LLM on MBPP and HumanEval+
• compare code models across 18 programming languages with MultiPL-E
• measure robustness of code LLMs with Recode
• run code generation evals in a sandboxed Docker container
• evaluate fill-in-the-middle code completion models

When to choose

• you need standardized pass@k evaluation for code generation models
• you want multi-language code benchmark coverage via MultiPL-E or HumanEvalPack
• you need reproducible, sandboxed execution of generated code

When to avoid

• you need to evaluate general-purpose (non-code) language models
• you want a hosted evaluation service rather than a local harness
• your model is not available on the Hugging Face Hub

Facets

framework · maturity active

machine-learning llm-inference benchmarking testing large-language-models machine-learning developer-tools programming-languages python code-generation llm-evaluation humaneval benchmark-suite huggingface linux docker gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="bigcode-project/bigcode-evaluation-harness")