openai/SWELancer-Benchmark resource

Health v2 · maintenance only

Flags: no_releases archived no_license

Adoption not part of the score

1431 stars · 135 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

SWE-Lancer is a benchmark dataset and evaluation harness measuring whether frontier LLMs can complete real-world freelance software engineering tasks worth up to $1 million in aggregate. The codebase has been merged into OpenAI's preparedness repository, so this repo now serves as a pointer to the active project.

Use cases

• evaluate llms on real-world software engineering tasks
• benchmark coding agents on freelance-style issues
• compare frontier models on paid software tasks
• run swe-lancer evaluations
• research llm coding capability
• measure ai performance on real github issues

When to choose

• you need a rigorous benchmark of LLM software engineering ability on real paid tasks
• you are researching how frontier models handle realistic freelance coding work
• you want reproducible evaluation harnesses from OpenAI's preparedness project

When to avoid

• you want an actively developed repo - use openai/preparedness instead
• you need a general-purpose coding assistant rather than an evaluation benchmark
• you lack the compute or API budget to run frontier model evaluations

Facets

dataset · maturity maintenance

benchmarking llm-inference machine-learning large-language-models artificial-intelligence developer-tools testing python cross-platform benchmark evaluation llm-evaluation software-engineering freelance-tasks openai swe-benchmark research docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="openai/SWELancer-Benchmark")