lm-sys/RouteLLM

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

5408 stars · 425 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

RouteLLM is a Python framework for serving and evaluating LLM routers that route queries between strong and weak models to cut costs. It provides an OpenAI-compatible client and server, pre-trained routers, and benchmarking tools.

Use cases

• reduce LLM API costs by routing simple queries to cheaper models
• serve an OpenAI-compatible endpoint that routes between GPT-4 and a smaller model
• benchmark and compare different LLM routing strategies
• calibrate cost/quality tradeoff thresholds for model routing
• route requests to local models instead of paid APIs

When to choose

• you pay for a strong LLM like GPT-4 but many queries could be handled by a cheaper model
• you want an OpenAI drop-in replacement with intelligent model routing
• you need to evaluate or research routing algorithms across benchmarks

When to avoid

• all your queries genuinely require top-tier model quality with no cost flexibility
• you use a single self-hosted model with no alternative to route to
• you need non-LLM request routing or generic API gateway features

Facets

framework · maturity active

llm-inference api-framework benchmarking http-server large-language-models machine-learning developer-tools apis python cloud llm-routing cost-optimization openai-compatible model-routing inference-optimization docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="lm-sys/RouteLLM")