open-compass/opencompass

Health v2 · maintenance only

Adoption not part of the score

7471 stars · 869 forks observed · 2026-09-24

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

OpenCompass is a Python-based evaluation platform and toolkit for benchmarking large language models across 100+ datasets, supporting models like Llama3, Mistral, Qwen, GPT-4, and Claude. It provides configurable evaluation pipelines, leaderboards (CompassRank), and a benchmark hub for comparing LLM quality.

Use cases

• evaluate llm performance across benchmarks
• compare open-source and proprietary language models
• run standardized llm evaluation suites
• benchmark my fine-tuned model on mmlu and other datasets
• build an llm leaderboard
• test chatgpt-style models on nlp tasks

When to choose

• you need reproducible, large-scale LLM benchmarking across many datasets
• you want to compare heterogeneous models (OpenAI APIs, HuggingFace checkpoints) under one framework
• you need a maintained, widely-adopted evaluation toolkit with an active community

When to avoid

• you only need quick ad-hoc prompt testing rather than systematic benchmarking
• you need evaluation of non-text modalities like images or audio
• you want a hosted no-setup evaluation service rather than a self-run framework

Facets

framework · maturity active

benchmarking machine-learning llm-inference nlp testing large-language-models machine-learning artificial-intelligence developer-tools python cli cross-platform llm-evaluation benchmarking-framework model-evaluation openai-api huggingface-models natural-language-processing linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="open-compass/opencompass")