meta-pytorch/gpt-fast

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

6249 stars · 575 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

A minimal (<1000 lines) PyTorch-native implementation of fast transformer text generation, demonstrating low-latency LLM inference with int8/int4 quantization, speculative decoding, and tensor parallelism. It is a reference codebase meant to be copied and forked rather than used as a framework, supporting LLaMA-family and Mixtral models on Nvidia and AMD GPUs.

Use cases

• run llama models fast with pure pytorch
• quantize llm to int8 or int4 for inference
• low latency single-user text generation on gpu
• learn how to optimize transformer inference
• speculative decoding example implementation
• tensor parallel inference across multiple gpus

When to choose

• you want minimal, hackable, dependency-light LLM inference code in pure PyTorch
• you need very low latency batch-size-1 generation on Nvidia or AMD GPUs
• you want a reference implementation to copy quantization or speculative decoding techniques into your own code

When to avoid

• you need a production framework with serving APIs, batching, or broad model support
• you want a maintained library with stable APIs - it is explicitly a demo/reference, not a framework
• you need CPU-only inference or non-GPU environments

Facets

library · maturity active

llm-inference machine-learning gpu-computing large-language-models deep-learning machine-learning performance python cross-platform transformer text-generation quantization speculative-decoding tensor-parallelism pytorch llama mixtral reference-implementation gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="meta-pytorch/gpt-fast")