NVIDIA/FasterTransformer

Health v2 · maintenance only

Adoption not part of the score

6447 stars · 933 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

NVIDIA's highly optimized C++/CUDA library for fast inference of Transformer-based models such as BERT, GPT, and encoder-decoder models, with TensorFlow, PyTorch, and Triton APIs. It leverages Tensor Cores with FP16/INT8/FP8 precision and supports tensor and pipeline parallelism, but development has moved to TensorRT-LLM.

Use cases

• run fast BERT inference on NVIDIA GPUs
• serve GPT models with tensor parallelism
• speed up transformer encoder and decoder inference
• use FP16 or INT8 transformer inference with Tensor Cores
• benchmark optimized transformer decoding performance
• integrate optimized transformer kernels into PyTorch or TensorFlow

When to choose

• you need maximum transformer inference throughput on NVIDIA GPUs and can use an older stack
• you want multi-GPU tensor or pipeline parallel inference for GPT-style models
• you need FP16/INT8 optimized BERT or GPT kernels in TensorFlow, PyTorch, or Triton

When to avoid

• you are starting a new project - use TensorRT-LLM instead, since FasterTransformer is no longer developed
• you need the latest model support or FP8/Hopper optimizations
• you do not have NVIDIA Volta or newer GPUs

Facets

library · maturity maintenance

llm-inference deep-learning machine-learning gpu-computing large-language-models deep-learning gpu-computing cpp python transformer bert gpt cuda tensor-cores inference-optimization tensor-parallelism pipeline-parallelism nvidia superseded-by-tensorrt-llm natural-language-processing linux gpu docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="NVIDIA/FasterTransformer")