vllm-project/semantic-router

Health v2 · maintenance only

Adoption not part of the score

5315 stars · 834 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

vLLM Semantic Router is an open-source programmable routing and control layer for building Mixture-of-Models systems across heterogeneous LLM infrastructure. It runs as an Envoy External Processor, exposing a stable OpenAI/Anthropic-compatible endpoint while selecting or composing model paths based on request signals, policies, and preferences.

Use cases

• route llm requests to the best model based on intent and complexity
• build a mixture-of-models gateway across cloud and local models
• cut llm inference costs by cascading cheap to expensive models
• add guardrails like jailbreak and pii detection to llm traffic
• route requests across heterogeneous gpu and edge compute
• enforce privacy policies so data stays within boundaries
• balance model quality, latency, and cost with slo-aware routing

When to choose

• you serve multiple llm models and want a single stable API endpoint with intelligent routing
• you need policy-driven routing with safety, caching, retrieval, and verification attached in the request path
• you want to optimize token spend, latency, and quality across a model fleet without hard-coding logic in apps
• you run kubernetes or envoy-based infrastructure and want routing as an observable control plane

When to avoid

• you serve a single model with no routing or policy needs
• you want a lightweight client-side router rather than an envoy-based gateway service
• you need a fully turnkey managed solution rather than a self-hosted, config-driven system

Facets

service · maturity active

llm-inference api-gateway middleware routing security caching rag monitoring large-language-models artificial-intelligence developer-tools self-hosted infrastructure-as-code go python self-hosted cloud mixture-of-models semantic-router envoy-extproc llm-routing ai-gateway guardrails model-cascades token-economy vllm ai-agents kubernetes docker linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="vllm-project/semantic-router")