princeton-nlp/MeZO

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1174 stars · 89 forks observed · 2026-09-25

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

MeZO is a memory-efficient zeroth-order optimizer that fine-tunes language models using only forward passes, with the same memory footprint as inference. It is a research implementation based on HuggingFace's Trainer, supporting full-parameter and parameter-efficient tuning like LoRA and prefix tuning.

Use cases

• fine-tune a 30B parameter language model on a single 80GB GPU
• train LLMs without backpropagation to reduce GPU memory usage
• optimize non-differentiable objectives like accuracy or F1
• apply LoRA or prefix tuning with zeroth-order optimization
• reproduce NeurIPS 2023 MeZO paper experiments on OPT and RoBERTa models

When to choose

• GPU memory is the bottleneck for fine-tuning large language models
• you need to optimize non-differentiable objectives
• you want parameter-efficient fine-tuning without storing optimizer states

When to avoid

• you need fast convergence on small models where Adam fine-tuning fits in memory
• you require a production-ready training framework rather than research code
• your task benefits strongly from gradient-based optimization

Facets

library · maturity stable

llm-training machine-learning deep-learning large-language-models machine-learning deep-learning python zeroth-order-optimization memory-efficient fine-tuning lora prefix-tuning research-code huggingface-trainer gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="princeton-nlp/MeZO")