# NVIDIA/TransformerEngine

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.

Repository: https://github.com/NVIDIA/TransformerEngine
Canonical: https://ross.abutalabs.com/products/transformerengine
Homepage: https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/index.html
Language: Python
License: Apache-2.0
License Family: permissive
Topics: cuda, deep-learning, gpu, machine-learning, python, pytorch, fp8, jax, fp4
Last push: 2026-08-26T20:40:56+00:00

## Health v2 (maintenance only)
Score: 99/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 99, release rhythm 98, longevity 100
- inputs: {"age_days": 1443, "days_push": 7, "days_rel": 19, "gap_med": 26, "n_releases_24m": 26}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 3504, forks 806 (observed 2026-08-28T04:08:07.595112+00:00)

## What it is
Transformer Engine is an NVIDIA library for accelerating Transformer model training and inference on NVIDIA GPUs using low-precision formats like FP8, MXFP8, and NVFP4. It provides optimized building blocks and an AMP-like API for PyTorch and JAX, plus a framework-agnostic C++ API.

## Use cases
- train large language models in fp8 precision on nvidia gpus
- speed up transformer training with lower memory usage
- use nvfp4 mixed precision on blackwell gpus
- integrate fp8 support into a custom deep learning framework via c++ api
- train mixture-of-experts models with fused kernels
- run reinforcement learning training with end-to-end fp8 precision

## When to choose
- you train or fine-tune large Transformer models on NVIDIA Hopper, Ada, or Blackwell GPUs
- you need FP8/FP4 low-precision training with minimal accuracy loss
- you use PyTorch or JAX and want drop-in accelerated Transformer layers
- you need MoE, tensor parallelism, and fused operations at scale

## When to avoid
- you train on non-NVIDIA GPUs or CPU-only hardware
- your models are small and don't benefit from low-precision GEMM speedups
- you need a framework-agnostic solution without CUDA dependencies

## Facets
- artifact type: library
- maturity: active
- function: machine-learning, deep-learning, llm-training, llm-inference, gpu-computing, transformers
- domain: deep-learning, large-language-models, machine-learning, gpu-computing
- platform: python
- tags: fp8, fp4, mixed-precision, pytorch, jax, nvidia, quantization, transformer-models, linux, gpu, cuda

## Member repositories
- NVIDIA/TransformerEngine (main) score 99

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:08:07.595112+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T18:35:46.693541+00:00, confidence not recorded.
  - readme: https://github.com/NVIDIA/TransformerEngine (fetched 2026-08-28T04:08:07.595112+00:00, sha aa556d531bf1)
  - homepage: https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/index.html (fetched 2026-08-29T09:29:50.639501+00:00, sha e55887053736)
  - site_page: https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/features/low_precision_training/fp8_blockwise_scaling/fp8_blockwise_scaling.html (fetched 2026-08-29T09:29:50.665812+00:00, sha 4fda13f1538d)
  - site_page: https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/installation.html (fetched 2026-08-29T09:29:50.648984+00:00, sha 2cc4cd93e7a5)
  - site_page: https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/faq.html (fetched 2026-08-29T09:29:50.651482+00:00, sha 6db783edf0b9)
  - site_page: https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/features/low_precision_training/index.html (fetched 2026-08-29T09:29:50.653649+00:00, sha 5e9b428b6da2)
  - site_page: https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/features/low_precision_training/introduction/introduction.html (fetched 2026-08-29T09:29:50.655544+00:00, sha b8b67c0dd282)
  - site_page: https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/features/low_precision_training/performance_considerations/performance_considerations.html (fetched 2026-08-29T09:29:50.658168+00:00, sha b75a4b9334f4)
  - site_page: https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/features/low_precision_training/fp8_current_scaling/fp8_current_scaling.html (fetched 2026-08-29T09:29:50.661133+00:00, sha 4cfdeaee2a47)
  - site_page: https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/features/low_precision_training/fp8_delayed_scaling/fp8_delayed_scaling.html (fetched 2026-08-29T09:29:50.663622+00:00, sha d753f5fd558e)
- Data as of 2026-08-30T08:39:29.467469+00:00.