davidmrau/mixture-of-experts

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1252 stars · 112 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A PyTorch re-implementation of the sparsely-gated Mixture-of-Experts layer from the paper 'Outrageously Large Neural Networks' by Shazeer et al. It provides a drop-in MoE module with noisy gating and auxiliary load-balancing loss, plus minimal training examples.

Use cases

• add a mixture-of-experts layer to a pytorch model
• reproduce the sparsely-gated MoE paper in pytorch
• train sparse expert networks with noisy gating
• experiment with MoE on CIFAR-10
• reference implementation for single-GPU MoE training

When to choose

• you want a simple, readable PyTorch MoE layer for research or teaching
• you need the original sparsely-gated MoE with noisy top-k gating and aux loss
• you need a lightweight single-GPU MoE baseline

When to avoid

• you need distributed or multi-GPU MoE training at scale (use FastMoE or DeepSpeed-MoE)
• you need production-grade, actively maintained MoE implementations
• you need modern token-routing MoE used in LLMs like Mixtral

Facets

library · maturity maintenance

machine-learning deep-learning machine-learning deep-learning large-language-models python mixture-of-experts pytorch sparse-gating noisy-gating research-code

Member repositories

For agents

markdown · JSON · MCP: product_card(name="davidmrau/mixture-of-experts")