jiaweizzhao/GaLore

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1700 stars · 167 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

GaLore is a PyTorch library implementing Gradient Low-Rank Projection for memory-efficient full-parameter training of large language models. It plugs into existing optimizers like AdamW with minimal code changes, reducing memory usage compared to low-rank adaptation methods such as LoRA.

Use cases

• train LLMs on consumer GPUs with limited memory
• reduce optimizer memory footprint during full-parameter fine-tuning
• fine-tune large language models without LoRA
• pretrain language models memory-efficiently
• apply gradient low-rank projection to existing optimizers
• run 7B model training on a single RTX 4090

When to choose

• you need full-parameter learning but lack GPU memory for standard optimizers
• you want a drop-in optimizer replacement with two lines of code
• you are fine-tuning or pretraining LLMs on limited hardware

When to avoid

• you need a battle-tested production training framework rather than a research algorithm
• your model is small enough that memory is not a constraint
• you require LoRA-style adapter export for inference

Facets

library · maturity active

llm-training machine-learning gpu-computing large-language-models deep-learning machine-learning python gradient-low-rank-projection memory-efficient-training optimizer lora-alternative pytorch gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="jiaweizzhao/GaLore")