Zefan-Cai/R-KV

Health v2 · maintenance only

Flags: no_releases no_license

Adoption not part of the score

1209 stars · 196 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

R-KV is a training-free, redundancy-aware KV cache compression method for reasoning LLMs, discarding repetitive tokens on-the-fly during decoding to retain ~full accuracy with only ~10% of the cache. It ships as a plug-and-play wrapper plus performance-hardened ports for vLLM and SGLang serving engines.

Use cases

• compress kv cache during llm chain-of-thought inference
• reduce gpu memory for reasoning model serving
• increase llm serving throughput under memory constraints
• run long self-reflection reasoning without cache blowup
• speed up rl rollout inference for reasoning models
• benchmark kv cache compression on gsm8k

When to choose

• serving reasoning models (CoT/self-reflection) whose long outputs bloat the KV cache
• you need near-lossless accuracy at 10-16% cache budget
• you use vLLM or SGLang and want a patch-based integration
• you want a training-free, plug-and-play compression wrapper

When to avoid

• compressing long prompts rather than long generated traces
• you need a permissively licensed dependency (no license file)
• non-reasoning workloads with short outputs where cache pressure is low
• you need a turnkey product rather than a research codebase

Facets

library · maturity active

llm-inference caching machine-learning benchmarking large-language-models machine-learning performance gpu-computing python kv-cache-compression reasoning-models vllm sglang chain-of-thought training-free inference-optimization research-paper gpu linux docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="Zefan-Cai/R-KV")