huggingface/optimum-nvidia

Health v2 · maintenance only

Adoption not part of the score

1037 stars · 103 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Optimum-NVIDIA is a Python library that bridges Hugging Face Transformers with NVIDIA TensorRT-LLM for highly optimized LLM inference on NVIDIA GPUs. It lets users run models like LLaMA 2 significantly faster by changing a single line in existing transformers code.

Use cases

• run LLM inference faster on NVIDIA GPUs
• accelerate text generation with TensorRT-LLM
• use fp8 quantization for LLM inference
• drop-in replacement for transformers pipelines
• serve LLaMA models at high tokens/second
• optimize Hugging Face models for Hopper and Ampere GPUs

When to choose

• you already use Hugging Face transformers and want faster inference on NVIDIA GPUs
• you need maximum LLM throughput with minimal code changes
• you run on Linux with CUDA 12.6 and supported NVIDIA hardware

When to avoid

• you are not on NVIDIA GPUs or Linux
• you need a stable, mature production library
• you want CPU or multi-vendor GPU inference

Facets

library · maturity experimental

llm-inference machine-learning sdk large-language-models machine-learning gpu-computing python tensorrt-llm nvidia hugging-face transformers fp8 text-generation linux gpu docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="huggingface/optimum-nvidia")