0xSero/turboquant

Health v2 · maintenance only

Flags: no_releases young

Adoption not part of the score

1739 stars · 195 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

TurboQuant is a Python library implementing near-optimal KV cache quantization for LLM inference, compressing keys to 3-bit and values to 2-bit using Triton kernels. It integrates with vLLM and supports dense and MoE architectures on NVIDIA GPUs.

Use cases

• reduce KV cache memory usage during LLM inference
• double the max context/token capacity of a vLLM deployment
• serve long-context LLMs on limited GPU VRAM
• quantize KV cache to 3-bit keys and 2-bit values
• speed up prefill and decode throughput for large context windows
• run MoE models with mixed full and linear attention layers more efficiently

When to choose

• you use vLLM and need to fit longer contexts or more concurrent requests in GPU memory
• you want drop-in KV cache compression with minimal quality loss
• you run inference on NVIDIA GPUs (RTX 3090/5090 class) with dense or MoE models

When to avoid

• your model uses only linear-attention layers, which TurboQuant cannot compress
• you need a non-vLLM inference engine without integration support
• you require permissive licensing since the project is GPL-3.0
• you need CPU-only or non-CUDA inference

Facets

library · maturity active

llm-inference gpu-computing caching machine-learning large-language-models machine-learning gpu-computing performance python kv-cache-quantization vllm triton-kernels inference-optimization moe gpu linux docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="0xSero/turboquant")