lmarena/arena-hard-auto resource

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1060 stars · 159 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Arena-Hard-Auto is an automatic benchmark for evaluating instruction-tuned LLMs using challenging real-world prompts and LLM-based judges (GPT-4.1, Gemini-2.5). It provides a cheap, fast approximation of Chatbot Arena human preference rankings with high correlation to LMArena results.

Use cases

• benchmark my instruction-tuned LLM automatically
• estimate how a model ranks on Chatbot Arena before deployment
• compare LLMs on hard open-ended prompts
• evaluate models on creative writing tasks
• run LLM-as-judge evaluations
• measure model separability on challenging queries

When to choose

• you need a fast, cheap proxy for human preference evaluation of chat LLMs
• you want to predict Chatbot Arena performance before release
• you need challenging real-world prompts including software engineering, math, and creative writing

When to avoid

• you need deterministic task-specific benchmarks like MMLU or code correctness tests
• you cannot afford API calls to commercial judge models
• you need human preference data rather than LLM-judge approximations

Facets

dataset · maturity active

benchmarking llm-inference large-language-models machine-learning artificial-intelligence testing python cli llm-evaluation llm-benchmark chatbot-arena model-ranking llm-judge leaderboard evaluation

Member repositories

For agents

markdown · JSON · MCP: product_card(name="lmarena/arena-hard-auto")