modelscope/evalscope

Health v2 · maintenance only

Adoption not part of the score

3312 stars · 467 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

EvalScope is a Python framework from the ModelScope community for evaluating large language models, vision-language models, embedding models, and AIGC models against built-in benchmarks like MMLU, C-Eval, and GSM8K. It also provides inference performance stress testing, agent-based evaluation with sandboxed tool use, multi-model arena battles, and a web dashboard for result visualization.

Use cases

• benchmark an LLM on MMLU and GSM8K
• stress test inference performance of a model service
• compare two models head-to-head in an arena
• evaluate a vision-language model
• evaluate RAG pipelines with rerankers and embeddings
• run agentic benchmarks like SWE-bench in a sandbox
• visualize evaluation results in a dashboard

When to choose

• you need a one-command evaluation pipeline for LLMs or VLMs via OpenAI-compatible APIs
• you want both quality benchmarks and inference performance metrics like TTFT and TPOT in one tool
• you need multi-backend support spanning OpenCompass, VLMEvalKit, and RAGEval
• you want pairwise model battles and interactive comparison reports

When to avoid

• you only need simple unit testing of application code rather than model evaluation
• you require a fully managed cloud evaluation service with no local setup
• your models are not reachable via supported API or local inference backends

Facets

framework · maturity active

benchmarking testing machine-learning llm-inference rag data-visualization large-language-models machine-learning artificial-intelligence developer-tools performance python cli cross-platform llm-evaluation vlm-evaluation stress-testing arena-mode model-benchmarking openai-api agent-evaluation

Member repositories

For agents

markdown · JSON · MCP: product_card(name="modelscope/evalscope")