EvolvingLMMs-Lab/lmms-eval

Health v2 · maintenance only

Flags: no_license

Adoption not part of the score

4377 stars · 647 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

lmms-eval is a unified Python framework for evaluating large multimodal models across text, image, video, and audio tasks, with 100+ benchmarks and 30+ model integrations. It emphasizes reproducible, efficient, and trustworthy evaluation pipelines for frontier model research.

Use cases

• evaluate a vision-language model on standard benchmarks
• compare multimodal model scores reproducibly across teams
• benchmark video understanding models
• run audio evaluation on large multimodal models
• measure LMM performance across image, video, and audio tasks
• set up a unified eval pipeline for model releases

When to choose

• you need reproducible multimodal benchmark numbers
• you want one toolkit covering text, image, video, and audio evals
• you are a research lab evaluating frontier LMMs at scale

When to avoid

• you only need text-only LLM evaluation (lm-eval-harness may suffice)
• you need a lightweight ad-hoc metric script rather than a full harness
• you require a commercial benchmarking service with hosted infrastructure

Facets

framework · maturity active

benchmarking machine-learning llm-inference testing large-language-models machine-learning computer-vision artificial-intelligence python cross-platform multimodal-evaluation vision-language-models vlm llm-evaluation benchmarks video-understanding audio-evaluation audio video gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="EvolvingLMMs-Lab/lmms-eval")