kvcache-ai/Mooncake

Health v2 · maintenance only

Adoption not part of the score

6398 stars · 1132 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Mooncake is a KVCache-centric disaggregated serving platform for LLM inference, originally built to serve Kimi by Moonshot AI. It separates prefill and decode clusters, pools KVCache across CPU, DRAM, and SSD resources via RDMA, and includes a Transfer Engine and distributed Mooncake Store.

Use cases

• serve LLMs with high throughput under SLO constraints
• build a distributed KVCache pool for vLLM or SGLang
• disaggregate prefill and decode for long-context workloads
• transfer KVCache between GPU nodes over RDMA
• speed up rollout data transfer in disaggregated reinforcement learning
• share KV cache across inference instances to avoid recomputation

When to choose

• you operate multi-node LLM inference at scale and need KVCache reuse
• you want prefill/decode disaggregation with vLLM, SGLang, or TRT-LLM
• you have RDMA-capable GPU clusters and long-context workloads
• you need a distributed cache backend for RL rollout or online training

When to avoid

• you serve small models on a single GPU without RDMA hardware
• you need a simple turnkey inference server rather than serving infrastructure
• your deployment lacks high-speed networking between nodes

Facets

service · maturity active

llm-inference caching gpu-computing streaming scheduling large-language-models artificial-intelligence microservices gpu-computing infrastructure-as-code cpp python self-hosted kvcache rdma disaggregation prefill-decode-separation vllm sglang kv-cache-pool llm-serving linux docker gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="kvcache-ai/Mooncake")