# bigcode-project/bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.

Repository: https://github.com/bigcode-project/bigcode-evaluation-harness
Canonical: https://ross.abutalabs.com/products/bigcode-evaluation-harness
Language: Python
License: Apache-2.0
License Family: permissive
Last push: 2025-07-22T13:18:09+00:00

## Health v2 (maintenance only)
Score: 38/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 33, release rhythm 8, longevity 100
- inputs: {"age_days": 1485, "days_push": 407, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1058, forks 259 (observed 2026-08-28T04:03:25.124476+00:00)

## What it is
A framework for evaluating autoregressive code generation language models on benchmarks like HumanEval, MBPP, MultiPL-E, and HumanEvalPack. It supports any Hugging Face Hub model, multi-GPU generation with accelerate, and sandboxed Docker execution for reproducible evaluation.

## Use cases
- evaluate a code generation model on HumanEval
- benchmark an LLM on MBPP and HumanEval+
- compare code models across 18 programming languages with MultiPL-E
- measure robustness of code LLMs with Recode
- run code generation evals in a sandboxed Docker container
- evaluate fill-in-the-middle code completion models

## When to choose
- you need standardized pass@k evaluation for code generation models
- you want multi-language code benchmark coverage via MultiPL-E or HumanEvalPack
- you need reproducible, sandboxed execution of generated code

## When to avoid
- you need to evaluate general-purpose (non-code) language models
- you want a hosted evaluation service rather than a local harness
- your model is not available on the Hugging Face Hub

## Facets
- artifact type: framework
- maturity: active
- function: machine-learning, llm-inference, benchmarking, testing
- domain: large-language-models, machine-learning, developer-tools, programming-languages
- platform: python
- tags: code-generation, llm-evaluation, humaneval, benchmark-suite, huggingface, linux, docker, gpu

## Member repositories
- bigcode-project/bigcode-evaluation-harness (main) score 38

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:03:25.124476+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T06:57:45.090663+00:00, confidence not recorded.
  - readme: https://github.com/bigcode-project/bigcode-evaluation-harness (fetched 2026-08-28T04:03:25.124476+00:00, sha a20fcc43054f)
- Data as of 2026-08-30T08:39:29.467469+00:00.