# bird-bench/BIRD-CRITIC-1

[NeurIPS 2025 Main] SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications

Repository: https://github.com/bird-bench/BIRD-CRITIC-1
Canonical: https://ross.abutalabs.com/products/bird-critic-1
Homepage: https://bird-critic.github.io/
Language: Python
License: MIT
License Family: permissive
Last push: 2026-03-23T23:05:33+00:00

## Health v2 (maintenance only)
Score: 53/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 73, release rhythm 35, longevity 41
- inputs: {"age_days": 582, "days_push": 163, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1098, forks 36 (observed 2026-08-28T04:03:34.855440+00:00)

## What it is
BIRD-CRITIC 1.0 (a.k.a. SWE-SQL) is a benchmark dataset of real-world SQL user issues for evaluating whether LLMs can diagnose and fix database problems across MySQL, PostgreSQL, SQL Server, Oracle, and BigQuery. It includes 600 development tasks, 200 held-out OOD tests, and an execution-based evaluation environment with a public leaderboard.

## Use cases
- evaluate llms on fixing real-world sql issues
- benchmark llm sql debugging across dialects
- test llm agents on postgresql sql problems
- compare models on a sql issue leaderboard
- research agentic sql issue resolution
- measure human-ai collaboration on sql troubleshooting

## When to choose
- you need a rigorous, execution-based benchmark for LLM SQL debugging
- you want cross-dialect coverage including PostgreSQL, MySQL, SQL Server, and Oracle
- you are benchmarking agentic or multi-turn SQL issue-solving pipelines

## When to avoid
- you need a simple text-to-SQL natural-language-to-query dataset rather than issue fixing
- you want a lightweight general coding benchmark not focused on SQL
- you cannot access gated ground-truth data requiring email request

## Facets
- artifact type: dataset
- maturity: active
- function: benchmarking, llm-inference, database, testing
- domain: databases, large-language-models, artificial-intelligence, developer-tools
- platform: python, cross-platform
- tags: sql-benchmark, text-to-sql, llm-evaluation, sql-debugging, swe-benchmark, leaderboard

## Member repositories
- bird-bench/BIRD-CRITIC-1 (main) score 53

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:03:34.855440+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T06:46:18.060376+00:00, confidence not recorded.
  - readme: https://github.com/bird-bench/BIRD-CRITIC-1 (fetched 2026-08-28T04:03:34.855440+00:00, sha d0eaa02e5087)
  - homepage: https://bird-critic.github.io/ (fetched 2026-08-29T12:49:44.804589+00:00, sha 1546f928a565)
- Data as of 2026-08-30T08:39:29.467469+00:00.