openai/frontier-evals resource

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1287 stars · 177 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A collection of benchmark evaluations from OpenAI for measuring frontier LLM capabilities, including PaperBench (AI paper replication), SWE-Lancer (freelance software engineering tasks), and EVMbench (smart contract security). Each eval is an isolated Python project managed with uv.

Use cases

• benchmark frontier LLM capabilities
• evaluate models on real freelance software engineering tasks
• test whether models can replicate state-of-the-art AI papers
• assess model performance on smart contract security tasks
• reproduce published OpenAI eval results
• compare model performance across capability benchmarks

When to choose

• you need standardized benchmarks for evaluating frontier LLMs
• you want to reproduce results from OpenAI's published eval papers
• you're researching model capabilities in coding, research replication, or security

When to avoid

• you need a general-purpose eval framework for building custom benchmarks
• you want lightweight model testing rather than heavyweight research-grade evals
• your use case is outside the included eval domains

Facets

dataset · maturity active

benchmarking testing artificial-intelligence large-language-models developer-tools security python cli llm-evals frontier-models paperbench swelancer evmbench model-capabilities ai-safety evaluation

Member repositories

For agents

markdown · JSON · MCP: product_card(name="openai/frontier-evals")