# openai/simple-evals

Repository: https://github.com/openai/simple-evals
Canonical: https://ross.abutalabs.com/products/simple-evals
Language: Python
License: MIT
License Family: permissive
Last push: 2026-04-22T22:16:18+00:00

## Health v2 (maintenance only)
Score: 60/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 78, release rhythm 35, longevity 62
- inputs: {"age_days": 874, "days_push": 133, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 4612, forks 506 (observed 2026-08-28T04:08:55.054851+00:00)

## What it is
A lightweight Python library from OpenAI for evaluating language models against benchmarks like MMLU, GPQA, MATH, HumanEval, SimpleQA, HealthBench, and BrowseComp. It hosts reference implementations used to transparently publish model accuracy numbers.

## Use cases
- evaluate an LLM on MMLU or GPQA benchmarks
- measure model factual accuracy with SimpleQA
- run HealthBench evaluations on a language model
- reproduce OpenAI's published benchmark numbers
- compare model performance on math and coding tasks
- benchmark a new model against o3 or o4-mini results

## When to choose
- you want lightweight, reference-quality LLM evaluation code
- you need to reproduce OpenAI's published benchmark scores
- you want to evaluate models on SimpleQA, HealthBench, or BrowseComp

## When to avoid
- you need a full-featured, actively maintained eval framework with new benchmarks
- you need evaluations updated for the latest models, since the repo is deprecated for new results
- you need a production evaluation pipeline with extensive integrations

## Facets
- artifact type: library
- maturity: maintenance
- function: machine-learning, benchmarking, llm-inference
- domain: large-language-models, machine-learning, developer-tools
- platform: python
- tags: llm-evaluation, benchmarks, openai, simpleqa, healthbench, browsecomp

## Member repositories
- openai/simple-evals (main) score 60

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:08:55.054851+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T18:19:42.744962+00:00, confidence not recorded.
  - readme: https://github.com/openai/simple-evals (fetched 2026-08-28T04:08:55.054851+00:00, sha e97a732b51b2)
- Data as of 2026-08-30T08:39:29.467469+00:00.