huggingface/optimum-quanto

Health v2 · maintenance only

Adoption not part of the score

1053 stars · 91 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Optimum Quanto is a PyTorch quantization backend for Hugging Face Optimum that quantizes model weights (int2/int4/int8/float8) and activations (int8/float8) with support for CUDA and MPS devices. It provides a seamless workflow from float to dynamic to static quantization, with serialization compatible with PyTorch weight_only and safetensors.

Use cases

• quantize a huggingface llm to int8 or int4
• reduce gpu memory usage of a pytorch model
• quantize pytorch models with float8 weights and activations
• run quantized inference on mps or cuda
• save and reload quantized models with safetensors
• speed up matrix multiplications with int8 int4 kernels

When to choose

• you need versatile eager-mode quantization for non-traceable PyTorch models
• you want to quantize Hugging Face models and serialize them with safetensors
• you need quantization on MPS (Apple Silicon) as well as CUDA

When to avoid

• you need production-ready quantization with active development - use bitsandbytes or torchAO instead
• you need torch.compile (dynamo) compatibility
• you need optimized kernels for all mixed matrix multiplications on all devices

Facets

library · maturity maintenance

llm-inference machine-learning sdk large-language-models machine-learning deep-learning python cross-platform pytorch quantization huggingface model-compression int8 int4 float8 safetensors gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="huggingface/optimum-quanto")