pytorch/ao

Health v2 · maintenance only

Flags: no_license

Adoption not part of the score

2957 stars · 604 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

TorchAO is a PyTorch-native library for model optimization through quantization and sparsity. It supports quantizing weights, gradients, optimizers, and activations using custom data types like int4, float8, and MX formats for both training and inference.

Use cases

• quantize llama models to int4 for faster inference
• speed up llm pre-training with float8
• apply quantization-aware training to recover accuracy
• reduce model memory footprint for serving
• sparsify model weights and gradients
• quantize models for vllm or huggingface serving

When to choose

• you use PyTorch and want native quantization for training or inference
• you need int4/float8/MX data types for LLM speedups
• you want QAT to recover quantization accuracy loss
• you serve models with vLLM, HF Transformers, or torchtune

When to avoid

• you need a framework-agnostic quantizer outside the PyTorch ecosystem
• you target CPU-only inference with no CUDA/XPU support
• you need post-training quantization for non-PyTorch models like ONNX

Facets

library · maturity active

machine-learning llm-inference llm-training gpu-computing deep-learning large-language-models machine-learning gpu-computing python quantization sparsity float8 int4 qat model-optimization pytorch inference-optimization training-optimization gpu cuda linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="pytorch/ao")