NVIDIA/TransformerEngine

Health v2 · maintenance only

Adoption not part of the score

3504 stars · 806 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Transformer Engine is an NVIDIA library for accelerating Transformer model training and inference on NVIDIA GPUs using low-precision formats like FP8, MXFP8, and NVFP4. It provides optimized building blocks and an AMP-like API for PyTorch and JAX, plus a framework-agnostic C++ API.

Use cases

• train large language models in fp8 precision on nvidia gpus
• speed up transformer training with lower memory usage
• use nvfp4 mixed precision on blackwell gpus
• integrate fp8 support into a custom deep learning framework via c++ api
• train mixture-of-experts models with fused kernels
• run reinforcement learning training with end-to-end fp8 precision

When to choose

• you train or fine-tune large Transformer models on NVIDIA Hopper, Ada, or Blackwell GPUs
• you need FP8/FP4 low-precision training with minimal accuracy loss
• you use PyTorch or JAX and want drop-in accelerated Transformer layers
• you need MoE, tensor parallelism, and fused operations at scale

When to avoid

• you train on non-NVIDIA GPUs or CPU-only hardware
• your models are small and don't benefit from low-precision GEMM speedups
• you need a framework-agnostic solution without CUDA dependencies

Facets

library · maturity active

machine-learning deep-learning llm-training llm-inference gpu-computing transformers deep-learning large-language-models machine-learning gpu-computing python fp8 fp4 mixed-precision pytorch jax nvidia quantization transformer-models linux gpu cuda

Member repositories

For agents

markdown · JSON · MCP: product_card(name="NVIDIA/TransformerEngine")