microsoft/Tutel

Health v2 · maintenance only

Adoption not part of the score

1016 stars · 112 forks observed · 2026-09-03

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Tutel is Microsoft's optimized Mixture-of-Experts (MoE) library for efficient training and inference of large language models, featuring dynamic parallelism/sparsity switching and low-precision (FP8/NVFP4/MXFP4) inference for MoE models like DeepSeek, Kimi, GLM, Qwen3, and GPT-OSS. It runs on NVIDIA and AMD GPUs (A100, H100, MI300 series) and integrates with PyTorch.

Use cases

• run mixture-of-experts LLM inference with FP8 or FP4 quantization
• train MoE models efficiently on multi-GPU clusters
• serve DeepSeek or Kimi models on AMD MI300X GPUs
• speed up MoE expert routing in PyTorch
• fit large MoE models into limited GPU memory with NVFP4
• benchmark MoE inference throughput against vLLM or SGLang

When to choose

• you need optimized MoE training or inference on NVIDIA or AMD GPUs
• you want to run very large MoE models (DeepSeek, Kimi, GLM) with FP4/FP8 quantization on limited VRAM
• you need dynamic parallelism switching for MoE workloads in PyTorch

When to avoid

• you need a general-purpose LLM serving stack with broad model support and tooling
• your models are dense (non-MoE) transformers
• you need a simple single-GPU inference solution without multi-GPU hardware

Facets

library · maturity active

llm-inference llm-training machine-learning gpu-computing large-language-models deep-learning machine-learning gpu-computing python cloud mixture-of-experts moe pytorch quantization fp8 nvfp4 mxfp4 deepseek distributed-training inference-optimization linux gpu docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="microsoft/Tutel")