ModelCloud/GPTQModel

Health v2 · maintenance only

Flags: no_license

Adoption not part of the score

1248 stars · 205 forks observed · 2026-09-03

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

GPTQModel is a production-ready Python toolkit for quantizing (compressing) large language models using GPTQ, AWQ, and related methods, with hardware-accelerated inference across NVIDIA, AMD, Intel, Huawei, and Apple hardware. It integrates with Hugging Face Transformers, vLLM, and SGLang for loading and serving quantized models.

Use cases

• quantize an LLM to 4-bit with GPTQ
• convert a Hugging Face model to AWQ format
• run quantized models on vLLM or SGLang
• compress large language models for consumer GPUs
• quantize models on AMD ROCm or Apple Silicon
• quantize MoE and multimodal models like DeepSeek or Qwen-VL

When to choose

• you need to quantize LLMs with GPTQ/AWQ and broad model architecture support
• you want hardware-accelerated inference across multiple GPU vendors
• you need integration with HF Transformers, vLLM, and SGLang

When to avoid

• you need quantization of non-LLM models like CNNs
• you want a GUI-based one-click quantization tool
• you only need to run pre-quantized models without quantizing your own

Facets

library · maturity active

machine-learning llm-inference llm-training gpu-computing large-language-models machine-learning deep-learning gpu-computing python windows quantization gptq awq vllm sglang huggingface model-compression inference-optimization linux macos gpu cuda

Member repositories

For agents

markdown · JSON · MCP: product_card(name="ModelCloud/GPTQModel")