vllm-project/guidellm

Health v2 · maintenance only

Adoption not part of the score

1547 stars · 219 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

GuideLLM is an SLO-aware benchmarking and evaluation platform for LLM inference deployments, simulating real-world workloads against OpenAI-compatible and vLLM-native servers. It generates configurable traffic patterns, supports real and synthetic multimodal datasets, and produces detailed latency and token-level reports for capacity planning.

Use cases

• benchmark my vllm server under production-like load
• measure ttft and inter-token latency distributions for an llm endpoint
• find the max request rate my llm deployment can handle before slo violations
• run load tests against an openai-compatible api
• plan capacity for llm inference servers
• evaluate llm serving performance with synthetic multimodal workloads

When to choose

• you need SLO-driven latency statistics (TTFT, ITL, e2e) for an LLM server
• you want reproducible traffic sweeps to find safe operating ranges
• you benchmark OpenAI-compatible or vLLM-native endpoints

When to avoid

• you need to benchmark non-LLM services or general web APIs
• you want model quality/accuracy evaluation rather than serving performance
• you need a GUI-based load testing tool

Facets

cli-tool · maturity active

benchmarking load-testing monitoring large-language-models machine-learning performance developer-tools python cli cross-platform llm-inference slo openai-compatible vllm capacity-planning synthetic-data

Member repositories

For agents

markdown · JSON · MCP: product_card(name="vllm-project/guidellm")