Accio-org/CommerceAgentBench resource

Health v2 · maintenance only

Flags: no_releases young

Adoption not part of the score

1192 stars · 85 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

CommerceAgentBench is a benchmark of 107 long-horizon agent tasks evaluated in high-fidelity, stateful, reproducible replicas of real online services (Shopify, Gmail, Stripe, Jira, etc.). It ships containerized task environments with deterministic or LLM-assisted verifiers and a public leaderboard for comparing model-harness pairs.

Use cases

• benchmark llm agents on real-world commerce workflows
• evaluate long-horizon agent task completion
• compare models on browser, CLI, and API/MCP tasks
• test agents against stateful replicas of real web services
• measure agent reliability with reproducible verifiers
• run cross-harness agent evaluations

When to choose

• you need reproducible, stateful evaluation of agents on multi-step business workflows
• you want to compare LLMs on browser, CLI, and API/MCP task execution
• you need deterministic or LLM-assisted grading of agent outcomes

When to avoid

• you need a lightweight single-turn QA benchmark
• your domain is unrelated to commerce or business workflows
• you want a benchmark that runs without container infrastructure

Facets

dataset · maturity active

benchmarking agent-framework testing mcp artificial-intelligence e-commerce large-language-models developer-tools python cli agent-evaluation llm-benchmark stateful-mocks long-horizon-tasks reproducibility leaderboard commerce-workflows containerized-eval ai-agents docker web-server

Member repositories

For agents

markdown · JSON · MCP: product_card(name="Accio-org/CommerceAgentBench")