evalplus/evalplus

Health v2 · maintenance only

Adoption not part of the score

1804 stars · 206 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

EvalPlus is a Python framework and benchmark suite for rigorously evaluating LLM-synthesized code, providing HumanEval+ and MBPP+ datasets with massively extended test suites plus EvalPerf for code efficiency evaluation. It supports many LLM backends (OpenAI, Anthropic, Gemini, local models) and is widely used by major LLM teams.

Use cases

• evaluate code generation quality of an LLM
• benchmark my model on HumanEval+ and MBPP+
• measure pass@1 for LLM-synthesized code
• compare code efficiency of LLM outputs
• run rigorous code correctness tests on generated Python
• evaluate long-context code understanding of models

When to choose

• you need rigorous, test-augmented evaluation of LLM code generation
• you want standardized benchmarks recognized by major LLM teams
• you need both correctness and efficiency evaluation of generated code

When to avoid

• you need to evaluate natural-language tasks rather than code
• you want a general-purpose unit testing framework for your own codebase
• you need production monitoring of live LLM applications

Facets

library · maturity active

testing benchmarking llm-inference data-generation large-language-models machine-learning developer-tools testing python cli code-evaluation humaneval-plus mbpp-plus evalperf program-synthesis llm-benchmark code-generation leaderboard docker linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="evalplus/evalplus")