# tatsu-lab/alpaca_eval

An automatic evaluator for instruction-following language models. Human-validated, high-quality, cheap, and fast.

Repository: https://github.com/tatsu-lab/alpaca_eval
Canonical: https://ross.abutalabs.com/products/alpaca_eval
Homepage: https://tatsu-lab.github.io/alpaca_eval/
Language: Jupyter Notebook
License: Apache-2.0
License Family: permissive
Topics: deep-learning, evaluation, foundation-models, instruction-following, large-language-models, leaderboard, nlp, rlhf
Last push: 2025-08-09T07:23:36+00:00

## Health v2 (maintenance only)
Score: 36/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 36, release rhythm 8, longevity 85
- inputs: {"age_days": 1196, "days_push": 389, "days_rel": 614, "gap_med": null, "n_releases_24m": 1}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 2012, forks 315 (observed 2026-08-28T04:06:05.197717+00:00)

## What it is
AlpacaEval is an LLM-based automatic evaluation framework for instruction-following language models, producing win rates against a GPT-4 baseline that correlate strongly (0.98) with human preferences from ChatBot Arena. It includes a public leaderboard, evaluation sets, GPT-4-based auto-annotators, and length-controlled win-rate metrics.

## Use cases
- evaluate an instruction-following LLM automatically
- compare chat LLMs on a leaderboard cheaply and fast
- measure win rates of my model against GPT-4
- build a custom LLM leaderboard
- create a new automatic evaluator for LLM outputs
- benchmark chat models without human annotation
- check how well my model follows user instructions

## When to choose
- you need a fast, cheap, human-correlated benchmark for chat/instruction-following LLMs
- you want to submit a model to a recognized community leaderboard
- you want to develop or validate new automatic evaluators or eval sets

## When to avoid
- you need comprehensive safety or capability testing beyond general instruction following
- you cannot or will not use OpenAI API credits for annotation
- you require a gold-standard human evaluation rather than an LLM-based proxy

## Facets
- artifact type: library
- maturity: active
- function: benchmarking, llm-inference, machine-learning, cli
- domain: large-language-models, machine-learning, developer-tools
- platform: python, cli, cross-platform
- tags: llm-evaluation, leaderboard, instruction-following, rlhf, gpt-4-annotator, win-rate, chatbot-benchmark, natural-language-processing

## Member repositories
- tatsu-lab/alpaca_eval (main) score 36

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:06:05.197717+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T03:01:03.142817+00:00, confidence not recorded.
  - readme: https://github.com/tatsu-lab/alpaca_eval (fetched 2026-08-28T04:06:05.197717+00:00, sha 7f3fc1e48756)
  - homepage: https://tatsu-lab.github.io/alpaca_eval/ (fetched 2026-08-29T10:41:00.162567+00:00, sha ef7dda755582)
- Data as of 2026-08-30T08:39:29.467469+00:00.