openai/human-eval resource
Code for the paper "Evaluating Large Language Models Trained on Code" observed · 2026-08-28
Health v2 · maintenance only
33/100
- Activity 2
- Release rhythm 35
- Longevity 100
Flags: no_releases
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-03. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 1884
- days_rel: n/a
- days_push: 593
- n_releases_24m: 0
Adoption not part of the score
3354 stars · 458 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded
HumanEval is a hand-written evaluation dataset of 164 Python programming problems with unit tests, plus a harness for measuring code-generation model performance via pass@k functional correctness. It accompanies OpenAI's paper 'Evaluating Large Language Models Trained on Code'.
Use cases
- benchmark a code-generation LLM on HumanEval
- compute pass@1 pass@10 pass@100 for code completions
- evaluate functional correctness of generated Python code
- compare code models against a standard coding benchmark
- run unit tests against model-generated completions
When to choose
- you need a standard, widely-cited benchmark for code generation models
- you want functional-correctness scoring with unit tests rather than string matching
- you are reproducing results from code LLM papers
When to avoid
- you need a sandboxed execution environment - the harness runs untrusted generated code and deliberately leaves sandboxing to you
- you need evaluation in languages other than Python
- you need a maintained multi-task benchmark suite - this is a single frozen dataset
Facets
dataset · maturity maintenance
testing benchmarking large-language-models machine-learning developer-tools programming-languages python cli cross-platform code-generation llm-evaluation humaneval pass-at-k benchmark-harness unit-tests evaluation
1 source
- readme: https://github.com/openai/human-eval · fetched 2026-08-28 · 607df2462951
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| openai/human-eval | main | 33 |
For agents
markdown · JSON · MCP: product_card(name="openai/human-eval")
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem