dvmazur/mixtral-offloading

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

2332 stars · 226 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A Python library enabling efficient inference of Mixtral-8x7B mixture-of-experts language models on limited hardware like Google Colab or consumer desktops. It combines mixed HQQ quantization with per-expert GPU offloading and LRU caching to fit the model into combined GPU and CPU memory.

Use cases

• run mixtral 8x7b on google colab
• run mixtral on a consumer gpu with limited vram
• quantize mixture-of-experts model for inference
• offload llm experts between cpu and gpu
• demo notebook for mixtral inference

When to choose

• you want to run Mixtral-8x7B on hardware with limited GPU memory
• you want a Colab-ready demo of quantized MoE inference
• you need HQQ-based mixed quantization with expert offloading

When to avoid

• you need a stable CLI or production inference server
• you need speculative expert prefetching or other techniques not yet implemented
• you need support for quantization methods beyond HQQ

Facets

library · maturity experimental

llm-inference machine-learning deep-learning large-language-models machine-learning deep-learning python cloud mixture-of-experts quantization hqq offloading mixtral colab pytorch gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="dvmazur/mixtral-offloading")