SWE-bench/SWE-bench resource

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

5719 stars · 953 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

SWE-bench is a benchmark and evaluation harness that tests whether large language models can resolve real-world GitHub issues by generating patches against actual codebases. It ships datasets (SWE-bench, Lite, Verified, Multimodal), a Docker-based reproducible evaluation harness, and official leaderboards.

Use cases

• evaluate llm on real github issues
• benchmark coding agents on software engineering tasks
• measure how well a model generates bug-fix patches
• compare coding assistants on a public leaderboard
• run reproducible docker-based code evaluations
• find a dataset of real software engineering problems for llm research

When to choose

• you need a standardized, reproducible benchmark for LLM software-engineering ability
• you want to compare your model or agent against published state-of-the-art results
• you need real-world issue/patch data rather than synthetic coding tasks

When to avoid

• you need a general coding benchmark for simple algorithmic problems (e.g., HumanEval-style)
• you cannot run Docker or containerized evaluations
• you want a lightweight unit-testing framework rather than an LLM evaluation suite

Facets

dataset · maturity active

benchmarking testing machine-learning llm-inference agent-framework large-language-models developer-tools machine-learning testing python llm-benchmark code-generation github-issues evaluation-harness leaderboard swe-agent patch-generation software-engineering docker linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="SWE-bench/SWE-bench")