THUDM/AgentBench resource

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

3695 stars · 276 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

AgentBench is a comprehensive benchmark for evaluating large language models as agents across multi-turn interactive tasks like database querying, OS interaction, web shopping, and knowledge graph reasoning. The current version integrates with AgentRL and uses function-calling style prompts with fully containerized Docker Compose deployment.

Use cases

• benchmark how well LLMs perform as agents on multi-turn tasks
• compare function-calling abilities of different LLMs
• evaluate agents on OS interaction and database tasks
• run containerized agent evaluation environments
• train agents with RL using AgentRL integration

When to choose

• you need standardized multi-environment evaluation of LLM agents
• you want reproducible, containerized agent benchmarks
• you are comparing function-calling performance across models

When to avoid

• you need a simple single-task eval harness
• you cannot run Docker or provide large Freebase data
• you only want static dataset benchmarks without interactive environments

Facets

dataset · maturity active

benchmarking agent-framework llm-inference testing large-language-models machine-learning developer-tools python cross-platform llm-agents evaluation function-calling reinforcement-learning leaderboard iclr ai-agents docker linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="THUDM/AgentBench")