PKU-YuanGroup/MoE-LLaVA

Health v2 · maintenance only

Adoption not part of the score

2322 stars · 139 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

MoE-LLaVA is an open-source Mixture-of-Experts based sparse large vision-language model, released with the MoE-Tuning training strategy from a TMM 2025 paper. It activates only top-k experts per token, achieving LLaVA-1.5-7B-level performance with roughly 3B activated parameters.

Use cases

• run a sparse vision-language model for image captioning
• build a multimodal chatbot that answers questions about images
• research mixture-of-experts training for LVLMs
• reduce inference cost of vision-language models via sparsity
• benchmark visual understanding and object hallucination
• fine-tune a MoE-based multi-modal model on custom data

When to choose

• you need an efficient sparse LVLM with low activated parameter count
• you are researching MoE architectures for multi-modal models
• you want a reproducible baseline from a peer-reviewed paper
• you need image understanding with strong hallucination resistance

When to avoid

• you need a production-grade, commercially supported multimodal API
• you lack GPU resources for training or inference
• you need video or audio understanding rather than images
• you want a plug-and-play model with broad ecosystem tooling

Facets

library · maturity active

machine-learning deep-learning llm-inference nlp computer-vision large-language-models computer-vision deep-learning artificial-intelligence python mixture-of-experts vision-language-model multimodal sparse-model moe-tuning research-code gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="PKU-YuanGroup/MoE-LLaVA")