IST-DASLab/gptq

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

2360 stars · 207 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Reference implementation of GPTQ, a one-shot post-training weight quantization method for large generative transformer models based on approximate second-order information. It quantizes models like OPT, BLOOM, and LLaMA to 2/3/4 bits with minimal accuracy loss and includes CUDA kernels for fast quantized inference.

Use cases

• quantize a large language model to 4-bit or 3-bit weights
• run a 175B-parameter model on a single GPU
• compress OPT or BLOOM models with weight grouping
• evaluate perplexity of quantized language models
• benchmark quantized matrix-vector product CUDA kernels
• apply act-order and true-sequential quantization to LLaMA

When to choose

• you need research-grade GPTQ quantization of OPT, BLOOM, or LLaMA models
• you want to reproduce the ICLR 2023 GPTQ paper results
• you need fast 3-bit CUDA kernels for generative inference

When to avoid

• you need a maintained production quantization library with broad model support
• you want easy integration with modern inference frameworks
• you need quantization of non-transformer architectures

Facets

library · maturity maintenance

llm-inference machine-learning benchmarking gpu-computing large-language-models deep-learning machine-learning gpu-computing python cli quantization post-training-quantization model-compression transformers cuda-kernels research-code iclr-2023 gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="IST-DASLab/gptq")