google/BIG-bench resource

Health v2 · maintenance only

Flags: no_releases archived

Adoption not part of the score

3247 stars · 618 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

BIG-bench is a collaborative benchmark with over 200 tasks for probing and measuring the capabilities of large language models. It includes a Python evaluation harness, a lightweight 24-task subset (BIG-bench Lite), and a public leaderboard for comparing model performance.

Use cases

• evaluate large language models on diverse tasks
• benchmark my LLM against other models
• find tasks that probe LLM reasoning capabilities
• compare language model performance on a leaderboard
• create custom benchmark tasks for language models
• measure how LLM capabilities scale with model size

When to choose

• you need a broad, community-accepted suite of tasks to evaluate LLMs
• you want to compare your model against published results on a standard leaderboard
• you are researching how language model capabilities extrapolate with scale

When to avoid

• you need a lightweight, fast evaluation of a single narrow capability
• you want a benchmark focused only on the newest frontier models or agentic tasks
• you need production inference tooling rather than evaluation

Facets

dataset · maturity maintenance

benchmarking machine-learning nlp llm-inference testing large-language-models machine-learning artificial-intelligence tutorials python cross-platform llm-evaluation benchmark language-models evaluation-suite leaderboard natural-language-processing

Member repositories

For agents

markdown · JSON · MCP: product_card(name="google/BIG-bench")