zhuzilin/ring-flash-attention

Health v2 · maintenance only

Adoption not part of the score

1061 stars · 102 forks observed · 2026-10-04

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A Python library implementing RingAttention on top of FlashAttention for distributed long-context transformer training. It provides varlen and batch APIs, zigzag/stripe variants, Llama3-style context parallelism, and a HuggingFace model adapter.

Use cases

• train LLMs with sequences longer than fit on one GPU
• enable context parallelism in HuggingFace training runs
• pack variable-length samples with ring attention
• use llama3-style context parallelism for varlen training
• reduce memory overhead of attention across multiple GPUs
• substitute flash attention in transformers models with ring attention

When to choose

• you need to train or fine-tune transformers on very long sequences across multiple GPUs
• you want drop-in ring/context-parallel attention compatible with flash_attention_2 and HuggingFace models
• you need varlen (packed samples) support with low memory overhead

When to avoid

• single-GPU training where flash attention alone suffices
• inference-only workloads without distributed setup
• non-CUDA or non-NCCL environments

Facets

library · maturity active

machine-learning llm-training gpu-computing deep-learning large-language-models gpu-computing python ring-attention flash-attention context-parallelism distributed-training transformers long-context varlen huggingface gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="zhuzilin/ring-flash-attention")