LMCache/LMCache

Health v2 · maintenance only

Adoption not part of the score

11460 stars · 1774 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

LMCache is a KV cache management layer for LLM inference that stores, compresses, and reuses KV caches across requests, sessions, and serving engine instances. It reduces time-to-first-token and improves throughput for long-context, agentic, multi-turn, and RAG workloads, integrating with engines like vLLM on NVIDIA and AMD hardware.

Use cases

• reduce time-to-first-token for LLM serving
• reuse KV cache across multi-turn conversations
• offload KV cache from GPU memory to tiered storage
• share KV cache across multiple vLLM instances
• speed up RAG and long-context inference
• monitor KV cache hit rates in production LLM serving

When to choose

• you serve LLMs with vLLM or compatible engines and want lower TTFT and higher throughput
• your workloads are multi-turn, agentic, or RAG-heavy with repeated prompt prefixes
• you need persistent, engine-independent KV cache storage across CPU, local disk, and remote backends
• you run on NVIDIA or AMD GPUs and want vendor-neutral cache infrastructure

When to avoid

• you need a general-purpose key-value database rather than LLM KV cache management
• your inference engine has no LMCache connector or KV cache integration support
• you run tiny single-user deployments where prefill cost is negligible

Facets

library · maturity active

caching llm-inference monitoring gpu-computing large-language-models machine-learning databases performance developer-tools python cloud kv-cache vllm prefill-reuse ttft-optimization tiered-storage cuda rocm inference-optimization gpu linux docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="LMCache/LMCache")