open-compass/VLMEvalKit

Health v2 · maintenance only

Adoption not part of the score

4359 stars · 755 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

VLMEvalKit is an open-source Python toolkit for evaluating large vision-language models (LMMs/LVLMs) across 80+ benchmarks with support for 220+ models. It provides one-command, generation-based evaluation with exact matching and LLM-based answer extraction, plus public leaderboards.

Use cases

• evaluate a vision-language model on VQA benchmarks
• benchmark multimodal LLMs like GPT-4V, LLaVA, Qwen-VL
• compare LMM performance across 80+ datasets
• run one-command evaluation of image and video understanding models
• generate leaderboard results for vision-language models
• evaluate models with thinking mode or long outputs

When to avoid

• you need to evaluate text-only LLMs without vision inputs
• you want a custom benchmark pipeline outside the supported dataset formats
• you need training or fine-tuning tooling rather than evaluation

Facets

library · maturity active

benchmarking machine-learning llm-inference computer-vision nlp machine-learning computer-vision large-language-models artificial-intelligence developer-tools python cross-platform vision-language-models multimodal-evaluation vqa llm-benchmarks model-evaluation leaderboard gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="open-compass/VLMEvalKit")