MoonshotAI/MoBA

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

2169 stars · 158 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

MoBA (Mixture of Block Attention) is a PyTorch implementation of a trainable block-sparse attention mechanism for long-context large language models, applying MoE-style top-k gating to select relevant KV blocks. It can seamlessly switch between full and sparse attention and is deployed in Kimi's long-context serving.

Use cases

• speed up long-context llm attention with sparse block attention
• train a transformer that switches between full and sparse attention
• implement mixture-of-experts style gating for attention
• reduce quadratic attention cost for long sequences
• research efficient attention mechanisms for llms

When to choose

• you are training or continue-training an llm for long-context workloads and want efficient sparse attention
• you need a flexible attention layer that can toggle between full and sparse modes
• you want a production-proven approach (used by Kimi) with FlashAttention integration

When to avoid

• you need a drop-in speedup for an existing pretrained model without retraining
• you only run inference and cannot continue-train the model
• your sequences are short enough that full attention cost is acceptable

Facets

library · maturity active

llm-training llm-inference machine-learning deep-learning large-language-models deep-learning machine-learning python sparse-attention mixture-of-experts flash-attention long-context transformer pytorch gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="MoonshotAI/MoBA")