mit-han-lab/smoothquant

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1677 stars · 206 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

SmoothQuant is a training-free post-training quantization (PTQ) method that enables W8A8 (8-bit weight, 8-bit activation) quantization for large language models by migrating quantization difficulty from activations to weights. This repository provides the reference Python implementation, supporting models like OPT, BLOOM, GLM, Llama, Falcon, Mistral, and Mixtral with up to 1.56x speedup and 2x memory reduction.

Use cases

• quantize an LLM to int8 weights and activations without retraining
• reduce GPU memory usage for serving large language models
• speed up LLM inference with W8A8 quantization
• serve a 530B parameter model on a single node
• evaluate quantization accuracy on Llama, Falcon, Mistral, or Mixtral models
• integrate activation-smoothing quantization into an inference engine

When to choose

• you need training-free W8A8 quantization of large language models with minimal accuracy loss
• you want to reduce memory footprint and hardware costs for LLM serving
• you are researching or benchmarking post-training quantization methods for LLMs

When to avoid

• you need 4-bit or lower-bit quantization rather than 8-bit
• you require a production-maintained integration - use TensorRT-LLM, ONNX Runtime, or Neural-Compressor which have adopted SmoothQuant
• you are quantizing small models where activation outliers are not a problem

Facets

library · maturity maintenance

llm-inference machine-learning deep-learning large-language-models machine-learning deep-learning python quantization post-training-quantization w8a8 int8 model-compression inference-optimization research-paper gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="mit-han-lab/smoothquant")