bird-bench/BIRD-CRITIC-1 resource

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1098 stars · 36 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

BIRD-CRITIC 1.0 (a.k.a. SWE-SQL) is a benchmark dataset of real-world SQL user issues for evaluating whether LLMs can diagnose and fix database problems across MySQL, PostgreSQL, SQL Server, Oracle, and BigQuery. It includes 600 development tasks, 200 held-out OOD tests, and an execution-based evaluation environment with a public leaderboard.

Use cases

• evaluate llms on fixing real-world sql issues
• benchmark llm sql debugging across dialects
• test llm agents on postgresql sql problems
• compare models on a sql issue leaderboard
• research agentic sql issue resolution
• measure human-ai collaboration on sql troubleshooting

When to choose

• you need a rigorous, execution-based benchmark for LLM SQL debugging
• you want cross-dialect coverage including PostgreSQL, MySQL, SQL Server, and Oracle
• you are benchmarking agentic or multi-turn SQL issue-solving pipelines

When to avoid

• you need a simple text-to-SQL natural-language-to-query dataset rather than issue fixing
• you want a lightweight general coding benchmark not focused on SQL
• you cannot access gated ground-truth data requiring email request

Facets

dataset · maturity active

benchmarking llm-inference database testing databases large-language-models artificial-intelligence developer-tools python cross-platform sql-benchmark text-to-sql llm-evaluation sql-debugging swe-benchmark leaderboard

Member repositories

For agents

markdown · JSON · MCP: product_card(name="bird-bench/BIRD-CRITIC-1")