openai/evals

Health v2 · maintenance only

Flags: no_releases no_license

Adoption not part of the score

19262 stars · 3060 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

OpenAI Evals is a Python framework for evaluating large language models and LLM-based systems, with an open-source registry of benchmark evals. It supports running existing evals and writing custom ones, including private evals built on your own data.

Use cases

• evaluate llm model quality across benchmarks
• compare how different model versions affect my use case
• build custom evals for my llm application
• run benchmark suite on openai models
• create private evals from my own data
• test llm system behavior before deploying

When to choose

• you are building with OpenAI models and want standardized evaluation
• you need a registry of ready-made LLM benchmarks
• you want to write custom evals in Python for your LLM workflows

When to avoid

• you need evaluation of non-OpenAI models without adaptation
• you want a GUI-only evaluation experience (the OpenAI Dashboard may suffice instead)
• you are not working with LLMs at all

Facets

framework · maturity active

testing benchmarking llm-inference prompt-engineering large-language-models machine-learning developer-tools artificial-intelligence python cli cross-platform llm-evaluation benchmarks openai eval-registry model-testing

Member repositories

For agents

markdown · JSON · MCP: product_card(name="openai/evals")