ovg-project/kvcached

Health v2 · maintenance only

Adoption not part of the score

1509 stars · 180 forks observed · 2026-10-02

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

kvcached is a Python library that brings OS-style virtual memory abstraction to KV cache management for LLM serving and training on shared GPUs. It decouples GPU virtual addressing from physical memory allocation, enabling elastic, demand-driven KV cache allocation and better GPU utilization under dynamic workloads.

Use cases

• share one GPU across multiple LLM serving instances
• reduce GPU memory waste from pre-allocated KV caches
• run vLLM or SGLang with elastic KV cache allocation
• co-serve online and offline LLM workloads on the same GPU
• improve GPU utilization under dynamic LLM traffic
• on-demand physical GPU memory allocation for KV caches

When to choose

• you serve multiple LLM instances on shared GPUs and need flexible memory allocation
• you use vLLM or SGLang and want elastic, demand-driven KV cache management
• you need to mix online and offline LLM workloads on the same hardware
• GPU memory utilization is a cost concern in your LLM serving stack

When to avoid

• you run a single LLM instance with plenty of dedicated GPU memory
• your inference engine is not vLLM or SGLang compatible
• you need a turnkey serving solution rather than a cache management layer
• you work on non-CUDA hardware without GPU virtual memory support

Facets

library · maturity active

llm-inference caching gpu-computing machine-learning large-language-models gpu-computing machine-learning performance python kv-cache gpu-sharing vllm sglang virtual-memory memory-management llm-serving inference-engine gpu linux cuda

Member repositories

For agents

markdown · JSON · MCP: product_card(name="ovg-project/kvcached")