mit-han-lab/llm-awq

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

3621 stars · 321 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

AWQ is a Python library implementing Activation-aware Weight Quantization for compressing large language models to low-bit (INT3/4) weights with efficient CUDA inference kernels. It includes a model zoo of pre-quantized LLMs and TinyChat, a turn-key on-device inference engine for LLMs and vision-language models on edge devices.

Use cases

• quantize llama models to 4-bit
• run llm inference on jetson edge device
• compress large language models to fit in limited gpu memory
• speed up llm inference with 4-bit weights
• run vision language model on low-power hardware
• quantize deepseek distilled models
• offline on-device chatbot inference

When to choose

• you need to run LLMs or VLMs on memory-constrained GPUs or edge devices like Jetson
• you want accurate INT4 weight-only quantization with fast CUDA kernels
• you want pre-quantized model checkpoints for popular LLM families

When to avoid

• you need quantization of non-LLM model types
• you need CPU-only inference without NVIDIA GPUs
• you need training or fine-tuning rather than inference compression

Facets

library · maturity active

llm-inference machine-learning gpu-computing benchmarking large-language-models machine-learning gpu-computing developer-tools python cross-platform quantization weight-quantization int4 model-compression inference-optimization edge-inference cuda-kernels vision-language-models gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="mit-han-lab/llm-awq")