ray-project/llmperf

Health v2 · maintenance only

Flags: archived

Adoption not part of the score

1127 stars · 203 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

LLMPerf is a Python library for benchmarking and validating the performance of LLM APIs. It runs load tests measuring inter-token latency and throughput, plus correctness tests, across providers like OpenAI-compatible endpoints, Anthropic, TogetherAI, and Hugging Face.

Use cases

• benchmark llm api latency and throughput
• load test an openai-compatible endpoint with concurrent requests
• compare performance of different llm providers
• measure inter-token latency for streaming llm responses
• validate llm api output correctness
• stress test llama models on hosted endpoints

When to choose

• you need standardized performance metrics (TTFT, inter-token latency, throughput) for LLM APIs
• you want to compare multiple LLM providers with consistent token counting
• you are evaluating inference endpoints before production deployment

When to avoid

• you need to benchmark self-hosted model hardware rather than APIs
• you need functional/quality evaluation of model outputs beyond basic correctness checks
• you need a actively developed tool with recent features

Facets

library · maturity maintenance

benchmarking llm-inference testing large-language-models performance machine-learning python cli llm-apis load-testing latency-measurement throughput openai-compatible

Member repositories

For agents

markdown · JSON · MCP: product_card(name="ray-project/llmperf")