lyogavin/airllm

Health v2 · maintenance only

Adoption not part of the score

32573 stars · 3447 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

AirLLM is a Python library that dramatically reduces LLM inference memory usage by streaming model layers (and MoE experts) from disk, enabling 70B models to run on a single 4GB GPU without quantization, distillation, or pruning. It supports models like Llama 3.1 405B, DeepSeek-V3, Qwen, and Kimi K3 on consumer hardware.

Use cases

• run a 70B LLM on a 4GB GPU
• run 405B Llama 3.1 on 8GB VRAM
• run DeepSeek-V3 671B on ~12GB of memory
• run large open-source models without quantization
• do LLM inference on consumer GPUs
• run large models on a Mac
• fine-tune with QLoRA on limited VRAM

When to choose

• you have limited GPU VRAM but lots of system RAM or disk
• you want to run huge models without quantizing them
• inference speed is not a priority
• you want to experiment with the largest open-source models on consumer hardware

When to avoid

• you need fast, production-grade inference throughput
• you have enough VRAM to fit the model normally
• you need batch serving of many concurrent requests
• you want the lowest latency possible

Facets

library · maturity active

llm-inference machine-learning deep-learning large-language-models machine-learning artificial-intelligence python windows memory-optimization model-layer-streaming low-vram llama qwen deepseek chinese-nlp lora qlora gpu linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="lyogavin/airllm")