thu-ml/SageAttention

Health v2 · maintenance only

Flags: prerelease_only

Adoption not part of the score

3684 stars · 491 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

SageAttention is a family of quantized attention kernels (INT8/FP8/FP4) that accelerate transformer inference 2-5x over FlashAttention with negligible accuracy loss. It works plug-and-play across language, image, and video generation models on Ampere, Ada, Hopper, and Blackwell GPUs.

Use cases

• speed up LLM inference attention computation
• accelerate video generation model inference
• quantize attention to int8 or fp8 for faster transformers
• replace flashattention with a faster drop-in kernel
• reduce inference latency for diffusion transformer models
• run ViT models faster on consumer GPUs

When to choose

• you need faster transformer inference on NVIDIA GPUs without retraining
• you run LLM, image, or video generation models and attention is the bottleneck
• you want plug-and-play acceleration compatible with torch.compile and distributed inference

When to avoid

• you need bit-exact attention outputs for precision-sensitive training or evaluation
• you target non-NVIDIA hardware like AMD or Apple GPUs
• your model uses attention variants unsupported by the kernels

Facets

library · maturity active

llm-inference machine-learning gpu-computing deep-learning large-language-models machine-learning gpu-computing image-processing python windows quantization attention-kernels flashattention inference-acceleration int8 fp8 fp4 triton video-generation diffusion-models plug-and-play video gpu cuda linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="thu-ml/SageAttention")