Zefan-Cai/KVCache-Factory

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1375 stars · 179 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

KVCache-Factory is a unified Python framework implementing many KV cache compression, retrieval, merging, and quantization methods for long-context LLM inference. It provides a single evaluation interface across methods like PyramidKV, SnapKV, H2O, Quest, MiniCache, and KIVI.

Use cases

• compress kv cache for long-context llm inference
• benchmark kv cache eviction methods on longbench
• reduce gpu memory for llama or mistral inference
• compare pyramidkv vs snapkv vs h2o
• quantize kv cache with kivi or kvquant
• offload kv cache to cpu for 70b models
• evaluate needle-in-a-haystack retrieval under cache compression

When to choose

• you need to experiment with or compare many KV cache compression methods under one interface
• you are running long-context inference on limited GPU memory
• you want reproducible LongBench-style evaluation of cache baselines

When to avoid

• you need production serving with high throughput rather than research evaluation
• your model architecture is not Llama or Mistral style
• you need broad coverage of the newest methods without checking runner support

Facets

library · maturity active

llm-inference machine-learning benchmarking large-language-models deep-learning machine-learning python kv-cache kv-cache-compression long-context inference-optimization pyramidkv quantization attention gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="Zefan-Cai/KVCache-Factory")