fla-org/native-sparse-attention

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1020 stars · 54 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Efficient Triton kernel implementations of Native Sparse Attention (NSA), a hardware-aligned, natively trainable sparse attention mechanism for long-context language models. It provides fused kernels for token compression, top-k block selection, and sliding-window attention, with support for training and variable-length inputs.

Use cases

• speed up long-context attention in transformer models
• pretrain language models with sparse attention
• implement natively trainable sparse attention kernels in Triton
• benchmark sparse attention performance on GPUs
• run attention with top-k block selection without materializing the attention matrix
• handle variable-length sequences with sparse attention

When to choose

• you need efficient, GPU-optimized sparse attention for long-context LLM training or inference
• you want to reproduce or build on the NSA paper (arXiv:2502.11089)
• you need fused Triton kernels combining selected and sliding-window attention
• you want to reduce pretraining compute without sacrificing model quality

When to avoid

• you need a general-purpose attention library with many attention variants
• your workloads are short-context where full attention is already fast enough
• you lack CUDA GPUs, since the kernels are Triton/CUDA-based
• you need production support or a stable API - the project is research-oriented

Facets

library · maturity active

machine-learning deep-learning llm-training gpu-computing benchmarking deep-learning large-language-models machine-learning gpu-computing python sparse-attention triton kernels long-context transformers attention-mechanism gpu linux cuda

Member repositories

For agents

markdown · JSON · MCP: product_card(name="fla-org/native-sparse-attention")