llm-d/llm-d

Health v2 · maintenance only

Adoption not part of the score

4164 stars · 706 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

llm-d is a Kubernetes-native distributed LLM inference serving stack that orchestrates model servers like vLLM and SGLang across clusters. It provides intelligent prefix-cache and load-aware routing, tiered KV-cache management, prefill/decode disaggregation, wide expert parallelism, and SLO-aware autoscaling, packaged as tested 'Well-Lit Path' Helm deployment recipes.

Use cases

• serve llm inference at scale on kubernetes
• route requests by prefix cache affinity
• deploy deepseek-r1 with wide expert parallelism
• autoscale llm inference pool based on slo
• offload kv cache to cpu and disk
• run vllm across multiple gpus in production
• reduce time to first token for multi-turn chat
• batch offline inference with openai-compatible api

When to choose

• you already run Kubernetes and want production-grade distributed LLM serving on GPUs, TPUs, or other accelerators
• you need LLM-aware load balancing, KV-cache reuse, or disaggregated prefill/decode beyond what a single vLLM/SGLang node provides
• you want vendor-neutral, engine-agnostic orchestration with tested deployment recipes and benchmarks

When to avoid

• you serve a single model on one node and vanilla vLLM or SGLang meets your needs
• you have no Kubernetes infrastructure and don't want to adopt it
• you need a simple managed API endpoint rather than operating your own inference cluster

Facets

framework · maturity active

llm-inference api-gateway load-testing caching monitoring deployment container-orchestration large-language-models machine-learning gpu-computing cloud-computing microservices infrastructure-as-code cloud self-hosted go python distributed-inference kv-cache vllm sglang intelligent-routing prefix-cache moe cncf-sandbox helm-charts autoscaling gateway-api-inference-extension containers kubernetes docker gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="llm-d/llm-d")