huggingface/lighteval

Health v2 · maintenance only

Adoption not part of the score

2527 stars · 544 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Lighteval is Hugging Face's all-in-one toolkit for evaluating large language models across multiple inference backends (Transformers, vLLM, SGLang, TGI, LiteLLM, Inference Endpoints, and more). It ships with 1000+ benchmark tasks and metrics, supports custom tasks and metrics, and saves detailed sample-by-sample results for debugging.

Use cases

• evaluate an LLM on MMLU or GSM8K benchmarks
• compare model performance across inference providers
• run LLM benchmarks on GPUs with vLLM
• create custom evaluation tasks and metrics for my model
• debug model outputs sample by sample from an eval run
• benchmark models served via an OpenAI-compatible API
• evaluate models on Hugging Face Inference Endpoints

When to choose

• you need a flexible, multi-backend LLM evaluation harness with a large benchmark catalog
• you want detailed, per-sample eval results for debugging model behavior
• you're in the Hugging Face ecosystem and want tight integration with Transformers, the Hub, and Inference Providers
• you need custom evaluation tasks or metrics

When to avoid

• you only need simple unit testing of code rather than model evaluation
• you need a hosted, no-setup evaluation service rather than a Python toolkit
• your evaluation targets are not language models (e.g., vision-only or classical ML models)

Facets

library · maturity active

benchmarking llm-inference cli developer-tools large-language-models machine-learning developer-tools python cli cross-platform llm-evaluation benchmarks mmlu hugging-face vllm inspect-ai evaluation-metrics model-evaluation natural-language-processing gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="huggingface/lighteval")