vllm-project/production-stack

Health v2 · maintenance only

Adoption not part of the score

2667 stars · 523 forks observed · 2026-10-14

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

vLLM Production Stack is a reference implementation for deploying vLLM LLM inference at scale on Kubernetes, distributed via Helm charts. It adds a request router with KV-cache-aware, prefix-aware, and load-aware routing, plus KV cache offloading via LMCache and Prometheus-based monitoring.

Use cases

• deploy vllm on kubernetes
• scale llm inference across multiple gpu nodes
• route llm requests with kv cache aware load balancing
• share kv cache across vllm instances with lmcach
• deploy openai-compatible llm api on aws or gcp
• monitor vllm inference metrics with prometheus
• set up disaggregated prefill for llm serving
• enable tool calling for llama models in production

When to choose

• you already use vLLM and need cluster-wide, multi-replica deployment on Kubernetes
• you want smart request routing (prefix/KV-cache/load aware) across vLLM instances
• you need KV cache offloading and sharing to improve throughput and fault tolerance
• you want an OpenAI-compatible LLM endpoint with built-in monitoring and autoscaling

When to avoid

• you only need a single vLLM instance on one machine without Kubernetes
• you use a different inference engine like TGI, SGLang, or TensorRT-LLM
• you cannot run a Kubernetes cluster or Helm
• you need a fully managed LLM API rather than self-hosted serving

Facets

framework · maturity active

llm-inference deployment monitoring caching container-orchestration infrastructure-as-code api-framework benchmarking tracing large-language-models artificial-intelligence cloud-computing self-hosted microservices python cloud self-hosted vllm helm-chart kv-cache lmcache request-routing openai-compatible-api llm-serving inference-serving load-balancing prefix-caching devops containers kubernetes docker gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="vllm-project/production-stack")