# NVIDIA/TensorRT-LLM

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

Repository: https://github.com/NVIDIA/TensorRT-LLM
Canonical: https://ross.abutalabs.com/products/tensorrt-llm
Homepage: https://nvidia.github.io/TensorRT-LLM
Language: Python
License: NOASSERTION
License Family: other
Topics: blackwell, cuda, moe, pytorch, llm-serving
Last push: 2026-08-27T00:12:30+00:00

## Health v2 (maintenance only)
Score: 84/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 99, release rhythm 68, longevity 79
- inputs: {"age_days": 1113, "days_push": 7, "days_rel": 135, "gap_med": 39.0, "n_releases_24m": 15}
- flags: no_license
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 14483, forks 2694 (observed 2026-08-28T04:11:07.099682+00:00)

## What it is
NVIDIA's TensorRT-LLM is a Python library and runtime for defining and serving Large Language Models with state-of-the-art inference optimizations on NVIDIA GPUs. It includes specialized kernels, an efficient orchestration runtime, and Python/C++ APIs for high-performance LLM deployment.

## Use cases
- serve LLMs with high throughput on NVIDIA GPUs
- run optimized inference for models like DeepSeek or Llama
- deploy an OpenAI-compatible LLM server with trtllm-serve
- benchmark and evaluate LLM serving performance
- run multimodal and visual generation model inference
- optimize MoE model serving on Blackwell GPUs
- run offline batch inference with a Pythonic LLM API

## When to choose
- you need maximum inference performance on NVIDIA GPUs
- you are serving LLMs in production at scale
- you want optimized kernels for MoE, quantization, or long-context models
- you need an OpenAI-compatible serving endpoint with metrics

## When to avoid
- you target non-NVIDIA hardware (AMD, Apple, CPU-only)
- you need a lightweight or simple inference setup
- you want to train or fine-tune models rather than serve them
- you need broad model coverage over raw speed

## Facets
- artifact type: library
- maturity: active
- function: llm-inference, gpu-computing, machine-learning, deep-learning, sdk
- domain: large-language-models, deep-learning, gpu-computing, artificial-intelligence
- platform: python, cpp
- tags: tensorrt, nvidia, inference-optimization, llm-serving, cuda-kernels, mixture-of-experts, multimodal, openai-compatible-api, linux, gpu, docker

## Member repositories
- NVIDIA/TensorRT-LLM (main) score 84

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:11:07.099682+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T17:12:32.070493+00:00, confidence not recorded.
  - readme: https://github.com/NVIDIA/TensorRT-LLM (fetched 2026-08-28T04:11:07.099682+00:00, sha 4d8cf6337183)
  - homepage: https://nvidia.github.io/TensorRT-LLM (fetched 2026-08-29T08:05:52.127738+00:00, sha 0c102a1fdc89)
  - registry_pypi: https://pypi.org/pypi/tensorrt-llm/json (fetched 2026-08-29T08:05:52.130793+00:00, sha 030645afb9f2)
- Data as of 2026-08-30T08:39:29.467469+00:00.