databricks/megablocks

Health v2 · maintenance only

Adoption not part of the score

1596 stars · 230 forks observed · 2026-10-11

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

MegaBlocks is a lightweight Python library for efficient training of mixture-of-experts (MoE) models, built around its dropless-MoE (dMoE) layers implemented with block-sparse operations. It integrates with Megatron-LM for data, expert, and pipeline parallel training and can also be used with frameworks like vLLM.

Use cases

• train mixture-of-experts language models efficiently
• avoid token dropping in MoE training without capacity_factor tuning
• accelerate transformer training with dropless MoE layers
• run distributed MoE training with Megatron-LM
• serve Mixtral-8x7B with vLLM using dMoE kernels
• use grouped GEMM MoE kernels on Hopper GPUs

When to choose

• you are training MoE transformer models in PyTorch and want high throughput without token dropping
• you already use Megatron-LM and want expert/data/pipeline parallel MoE support
• you need efficient block-sparse or grouped GEMM MoE kernels on NVIDIA GPUs

When to avoid

• you only need MoE inference without training
• you are not working with GPU-accelerated PyTorch training
• you need a general-purpose deep learning framework rather than MoE-specific layers

Facets

library · maturity active

machine-learning llm-training gpu-computing machine-learning deep-learning large-language-models python mixture-of-experts moe dropless-moe block-sparse megatron-lm pytorch distributed-training gpu docker linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="databricks/megablocks")