NVIDIA/TensorRT-LLM

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way. observed · 2026-08-28

github.com/NVIDIA/TensorRT-LLM · homepage · Python · NOASSERTION (other) observed · 2026-08-28

Health v2 · maintenance only

84/100

Activity 99
Release rhythm 68
Longevity 79

Flags: no_license

How is this computed?

round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-03. Adoption (stars, forks) is never an input.

gap_med: 39.0
age_days: 1113
days_rel: 135
days_push: 7
n_releases_24m: 15

Full methodology

Adoption not part of the score

14483 stars · 2694 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

NVIDIA's TensorRT-LLM is a Python library and runtime for defining and serving Large Language Models with state-of-the-art inference optimizations on NVIDIA GPUs. It includes specialized kernels, an efficient orchestration runtime, and Python/C++ APIs for high-performance LLM deployment.

Use cases

serve LLMs with high throughput on NVIDIA GPUs
run optimized inference for models like DeepSeek or Llama
deploy an OpenAI-compatible LLM server with trtllm-serve
benchmark and evaluate LLM serving performance
run multimodal and visual generation model inference
optimize MoE model serving on Blackwell GPUs
run offline batch inference with a Pythonic LLM API

When to choose

you need maximum inference performance on NVIDIA GPUs
you are serving LLMs in production at scale
you want optimized kernels for MoE, quantization, or long-context models
you need an OpenAI-compatible serving endpoint with metrics

When to avoid

you target non-NVIDIA hardware (AMD, Apple, CPU-only)
you need a lightweight or simple inference setup
you want to train or fine-tune models rather than serve them
you need broad model coverage over raw speed

Facets

library · maturity active

llm-inference gpu-computing machine-learning deep-learning sdk large-language-models deep-learning gpu-computing artificial-intelligence python cpp tensorrt nvidia inference-optimization llm-serving cuda-kernels mixture-of-experts multimodal openai-compatible-api linux gpu docker

3 sources

readme: https://github.com/NVIDIA/TensorRT-LLM · fetched 2026-08-28 · 4d8cf6337183
homepage: https://nvidia.github.io/TensorRT-LLM · fetched 2026-08-29 · 0c102a1fdc89
registry_pypi: https://pypi.org/pypi/tensorrt-llm/json · fetched 2026-08-29 · 030645afb9f2

Member repositories

Repository	Role	Health v2
NVIDIA/TensorRT-LLM	main	84

For agents

markdown · JSON · MCP: product_card(name="NVIDIA/TensorRT-LLM")

Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem