tonbistudio/turboquant-pytorch

Health v2 · maintenance only

Flags: no_releases young

Adoption not part of the score

1040 stars · 138 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A from-scratch PyTorch implementation of Google's TurboQuant (ICLR 2026) algorithm for compressing LLM key-value caches, including an improved V3 variant that removes the paper's QJL component. It provides quantized KV cache utilities with verified compression/attention-fidelity benchmarks.

Use cases

• compress llm kv cache to fit longer contexts in gpu memory
• quantize key-value cache to 3-4 bits in pytorch
• reproduce turboquant paper results
• run long-context llm inference with lower memory usage
• benchmark kv cache compression against attention fidelity
• implement vector quantization for transformer caches

When to choose

• you need KV cache compression for LLM inference in PyTorch
• you want to reproduce or extend the TurboQuant paper
• you need verified, benchmarked quantization with honest results reporting

When to avoid

• you need production-ready, drop-in KV cache compression for a serving stack
• you expect the advertised 5x compression at 3-bit to work for generation - tests show 3-bit output is broken
• you use non-NVIDIA GPUs or non-PyTorch frameworks

Facets

library · maturity active

machine-learning deep-learning llm-inference caching compression large-language-models deep-learning machine-learning performance python windows cross-platform kv-cache-compression vector-quantization pytorch quantization llm-optimization research-reproduction gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="tonbistudio/turboquant-pytorch")