TheTom/turboquant_plus

Health v2 · maintenance only

Flags: no_releases young

Adoption not part of the score

7006 stars · 925 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

TurboQuant+ is a Python reference implementation of the TurboQuant KV cache compression method (ICLR 2026), using PolarQuant codebooks and Walsh-Hadamard rotation to compress transformer KV caches 3.8-6.4x. It serves as the research home with benchmarks and validation results, with stable pieces upstreamed into engines like vLLM and llama.cpp.

Use cases

• compress kv cache for long-context llm inference
• run 100b model on limited memory via cache quantization
• benchmark kv cache quantization methods
• reduce gpu memory usage for transformer inference
• research extreme kv cache compression
• speed up llm decode at long context

When to choose

• you need the full PolarQuant codec or asymmetric K/V policies not yet upstreamed
• you want the reference implementation and benchmark data for research
• you want to run very large models at long context on consumer hardware like a MacBook

When to avoid

• you just want production KV compression in a datacenter engine - use the merged vLLM flag instead
• you need a turnkey inference server - this is a research library, not an inference engine
• you need non-transformer model compression

Facets

library · maturity active

llm-inference machine-learning benchmarking gpu-computing large-language-models deep-learning machine-learning performance python windows cross-platform kv-cache-compression quantization polarquant walsh-hadamard-transform inference-optimization research-code memory-optimization gpu macos linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="TheTom/turboquant_plus")