EleutherAI/lm-evaluation-harness

Health v2 · maintenance only

Adoption not part of the score

14198 stars · 3671 forks observed · 2026-10-13

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

A Python framework for few-shot evaluation of language models across hundreds of standard benchmarks (HellaSwag, MMLU, BIG-Bench, etc.). It supports HuggingFace, vLLM, SGLang, and API-hosted models with configurable tasks via YAML and a refactored CLI.

Use cases

• evaluate an LLM on standard benchmarks like MMLU or HellaSwag
• compare language models on few-shot tasks
• run the Open LLM Leaderboard task suite
• benchmark a HuggingFace or vLLM model
• evaluate models served via OpenAI-compatible APIs
• create custom evaluation tasks with YAML configs
• measure chain-of-thought reasoning performance

When to choose

• you need reproducible, standardized LLM benchmark scores
• you want to evaluate models across many backends (HF, vLLM, SGLang, APIs)
• you need configurable few-shot prompting and custom tasks
• you're submitting or reproducing Open LLM Leaderboard results

When to avoid

• you need broad multimodal (text+image) evaluation coverage - consider lmms-eval
• you want training or fine-tuning rather than evaluation
• you need a hosted no-code evaluation service

Facets

framework · maturity active

machine-learning llm-inference benchmarking testing cli large-language-models machine-learning developer-tools python cli windows llm-evaluation benchmarks few-shot huggingface vllm open-llm-leaderboard model-evaluation natural-language-processing gpu linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="EleutherAI/lm-evaluation-harness")