tspeterkim/flash-attention-minimal

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1179 stars · 115 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A minimal ~100-line CUDA re-implementation of the Flash Attention forward pass, loadable as a PyTorch extension. It is designed as an educational resource for CUDA beginners, following the notation of the original Flash Attention paper.

Use cases

• learn how flash attention works in CUDA
• understand shared memory tiling for attention
• benchmark flash attention vs manual attention
• study a minimal CUDA kernel for transformer attention
• starting point for writing custom attention kernels

When to choose

• you want to learn how Flash Attention is implemented in CUDA
• you need a small readable reference implementation of the attention forward pass
• you are teaching or studying GPU kernel design for transformers

When to avoid

• you need a backward pass for training
• you need production performance or float16/tensor core support
• you need long sequence attention at scale
• you want a maintained, feature-complete attention library

Facets

library · maturity experimental

machine-learning deep-learning gpu-computing deep-learning large-language-models gpu-computing developer-tools python flash-attention cuda-kernel educational pytorch-extension transformer-attention cuda gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="tspeterkim/flash-attention-minimal")