THUDM/LongBench resource

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1229 stars · 138 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

LongBench is a benchmark suite (v1 and v2) for evaluating large language models on long-context understanding and reasoning tasks, with contexts from 8k to 2M words across six task categories. It includes 503 challenging multiple-choice questions in v2, plus datasets, evaluation code, and a public leaderboard.

Use cases

• evaluate llm long-context understanding
• benchmark models on long document qa
• compare llms on 128k+ token contexts
• test long-context reasoning ability of a model
• measure effect of inference-time compute on long-context tasks
• find leaderboard results for long-context benchmarks

When to choose

• you need a standardized, reliable benchmark for long-context LLM evaluation
• you want multiple-choice scoring for reproducible comparisons
• you need contexts spanning 8k to 2M words across diverse task types

When to avoid

• you need open-ended generation quality evaluation rather than multiple-choice accuracy
• you need short-context or general-purpose benchmarks
• you need a training dataset rather than an evaluation benchmark

Facets

dataset · maturity active

benchmarking llm-inference nlp large-language-models machine-learning python long-context evaluation llm-benchmark multiple-choice leaderboard natural-language-processing algorithms

Member repositories

For agents

markdown · JSON · MCP: product_card(name="THUDM/LongBench")