pjlab-sys4nlp/llama-moe

Health v2 · maintenance only

Adoption not part of the score

1001 stars · 61 forks observed · 2026-09-03

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

LLaMA-MoE is a Python toolkit and model series for building Mixture-of-Experts (MoE) language models from dense LLaMA models via expert construction and continual pre-training. It provides multiple expert partitioning methods, gating strategies, and efficient training utilities with FlashAttention-v2 and streaming data loading.

Use cases

• convert a dense LLaMA model into a mixture-of-experts model
• continually pre-train an MoE LLM on my own dataset
• compare expert construction methods like clustering or co-activation graph partitioning
• train a small affordable sparse MoE model with top-K gating
• fine-tune an MoE language model with supervised fine-tuning
• monitor gate load and balance loss during MoE training

When to choose

• you want to build or study MoE models derived from existing dense LLMs
• you need a lightweight ~3.5B activated-parameter open model for research or deployment
• you want reproducible code from a published EMNLP 2024 paper on MoE construction

When to avoid

• you need a production inference server rather than a training/research toolkit
• you lack multi-GPU resources for continual pre-training
• you want a general-purpose LLM fine-tuning framework without MoE specifics

Facets

library · maturity active

llm-training machine-learning deep-learning sdk large-language-models machine-learning deep-learning python mixture-of-experts llama continual-pre-training expert-partitioning moe model-training huggingface natural-language-processing gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="pjlab-sys4nlp/llama-moe")