flashinfer-ai/flashinfer

Health v2 · maintenance only

Adoption not part of the score

6252 stars · 1340 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

FlashInfer is a GPU kernel library and kernel generator for LLM inference, providing unified APIs for attention, GEMM, and MoE operations with multiple backends (FlashAttention-2/3, cuDNN, CUTLASS, TensorRT-LLM). It supports paged KV-cache, FP8/FP4 low-precision compute, CUDA Graphs, and GPUs from Turing through Blackwell.

Use cases

• speed up LLM inference serving on NVIDIA GPUs
• efficient attention kernels with paged KV-cache for dynamic batching
• fused mixture-of-experts kernels for MoE models
• FP8 and FP4 quantized attention and GEMM
• accelerate decode and prefill in vLLM or custom serving engines
• shared-prefix cascade attention for memory-efficient batching
• MLA attention kernels for DeepSeek models

When to choose

• you need state-of-the-art GPU kernel performance for LLM serving
• you're building or tuning an inference engine like vLLM or SGLang
• you need low-precision (FP8/FP4) attention, GEMM, or MoE on modern NVIDIA GPUs
• you want backend flexibility with automatic hardware-aware kernel selection

When to avoid

• you're running inference on CPUs or non-NVIDIA accelerators without supported backends
• you just need a simple high-level inference API rather than kernel-level control
• your workload is small-scale where kernel optimization overhead isn't worth it

Facets

library · maturity active

llm-inference gpu-computing machine-learning deep-learning benchmarking large-language-models deep-learning gpu-computing machine-learning developer-tools python cuda-kernels attention mixture-of-experts kv-cache fp8 fp4 jit-compilation vllm inference-serving pytorch gpu linux cuda

Member repositories

For agents

markdown · JSON · MCP: product_card(name="flashinfer-ai/flashinfer")