datacurve-ai/deep-swe resource

Health v2 · maintenance only

Flags: no_releases young

Adoption not part of the score

1499 stars · 99 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks drawn from active open-source repositories, used to measure frontier coding agents across TypeScript, Go, Python, JavaScript, and Rust. It ships with isolated Docker environments, program-based verifiers, and a public leaderboard of model results.

Use cases

• evaluate coding agents on realistic long-horizon engineering tasks
• compare frontier LLMs on software engineering benchmarks
• benchmark a new model against Claude, GPT, Gemini, and DeepSeek results
• run sandboxed agent evaluations with network allowlists
• measure pass@1 and cost efficiency of coding agents

When to choose

• you need a rigorous, verifiable benchmark for coding agents
• you want up-to-date leaderboard comparisons across many LLMs
• you need isolated, reproducible eval environments with held-out tests

When to avoid

• you need a lightweight quick eval rather than full Docker-based runs
• you want short single-function coding tasks instead of long-horizon work
• you cannot run Docker or lack API access to frontier models

Facets

dataset · maturity active

benchmarking testing agent-framework developer-tools artificial-intelligence large-language-models developer-tools testing python cross-platform coding-agents benchmark llm-evaluation software-engineering-tasks leaderboard harbor-format ai-agents docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="datacurve-ai/deep-swe")