# huggingface/lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Repository: https://github.com/huggingface/lighteval
Canonical: https://ross.abutalabs.com/products/lighteval
Homepage: https://huggingface.co/docs/lighteval/en/index
Language: Python
License: MIT
License Family: permissive
Topics: evaluation, evaluation-framework, evaluation-metrics, huggingface
Last push: 2026-08-11T13:10:37+00:00

## Health v2 (maintenance only)
Score: 77/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 97, release rhythm 58, longevity 67
- inputs: {"age_days": 950, "days_push": 22, "days_rel": 282, "gap_med": 29, "n_releases_24m": 12}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 2527, forks 544 (observed 2026-08-28T04:06:58.648687+00:00)

## What it is
Lighteval is Hugging Face's all-in-one toolkit for evaluating large language models across multiple inference backends (Transformers, vLLM, SGLang, TGI, LiteLLM, Inference Endpoints, and more). It ships with 1000+ benchmark tasks and metrics, supports custom tasks and metrics, and saves detailed sample-by-sample results for debugging.

## Use cases
- evaluate an LLM on MMLU or GSM8K benchmarks
- compare model performance across inference providers
- run LLM benchmarks on GPUs with vLLM
- create custom evaluation tasks and metrics for my model
- debug model outputs sample by sample from an eval run
- benchmark models served via an OpenAI-compatible API
- evaluate models on Hugging Face Inference Endpoints

## When to choose
- you need a flexible, multi-backend LLM evaluation harness with a large benchmark catalog
- you want detailed, per-sample eval results for debugging model behavior
- you're in the Hugging Face ecosystem and want tight integration with Transformers, the Hub, and Inference Providers
- you need custom evaluation tasks or metrics

## When to avoid
- you only need simple unit testing of code rather than model evaluation
- you need a hosted, no-setup evaluation service rather than a Python toolkit
- your evaluation targets are not language models (e.g., vision-only or classical ML models)

## Facets
- artifact type: library
- maturity: active
- function: benchmarking, llm-inference, cli, developer-tools
- domain: large-language-models, machine-learning, developer-tools
- platform: python, cli, cross-platform
- tags: llm-evaluation, benchmarks, mmlu, hugging-face, vllm, inspect-ai, evaluation-metrics, model-evaluation, natural-language-processing, gpu

## Member repositories
- huggingface/lighteval (main) score 77

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:06:58.648687+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T02:25:24.678832+00:00, confidence not recorded.
  - readme: https://github.com/huggingface/lighteval (fetched 2026-08-28T04:06:58.648687+00:00, sha 28d4b57d8de4)
  - homepage: https://huggingface.co/docs/lighteval/en/index (fetched 2026-08-29T10:08:00.213414+00:00, sha 7e55e44e81d2)
  - site_page: https://huggingface.co/docs (fetched 2026-08-29T10:08:00.222394+00:00, sha bdec26667b98)
  - site_page: https://huggingface.co/docs/lighteval/en/installation (fetched 2026-08-29T10:08:00.226418+00:00, sha a165e044edd3)
  - site_page: https://huggingface.co/docs/lighteval/en/quicktour (fetched 2026-08-29T10:08:00.228827+00:00, sha 8420716d3848)
  - site_page: https://huggingface.co/docs/lighteval/en/inspect-ai (fetched 2026-08-29T10:08:00.231319+00:00, sha 0e6c8d774300)
  - site_page: https://huggingface.co/docs/lighteval/en/saving-and-reading-results (fetched 2026-08-29T10:08:00.233323+00:00, sha f808ab67589b)
  - site_page: https://huggingface.co/docs/lighteval/en/caching (fetched 2026-08-29T10:08:00.235171+00:00, sha a27b526c57bd)
  - site_page: https://huggingface.co/docs/lighteval/en/using-the-python-api (fetched 2026-08-29T10:08:00.236968+00:00, sha 92633e7b2be8)
  - registry_pypi: https://pypi.org/pypi/lighteval/json (fetched 2026-08-29T10:08:00.238719+00:00, sha 697e567aa042)
- Data as of 2026-08-30T08:39:29.467469+00:00.