# triton-inference-server/server

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

Repository: https://github.com/triton-inference-server/server
Canonical: https://ross.abutalabs.com/products/server
Homepage: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html
Language: Python
License: BSD-3-Clause
License Family: permissive
Topics: inference, gpu, machine-learning, deep-learning, cloud, datacenter, edge
Last push: 2026-08-26T20:29:13+00:00

## Health v2 (maintenance only)
Score: 98/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 99, release rhythm 95, longevity 100
- inputs: {"age_days": 2890, "days_push": 7, "days_rel": 35, "gap_med": 28.0, "n_releases_24m": 23}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 10939, forks 1830 (observed 2026-08-28T04:10:44.752594+00:00)

## What it is
NVIDIA Triton Inference Server is an open-source inference serving software that deploys AI models from multiple frameworks (TensorRT, PyTorch, ONNX, OpenVINO, Python, vLLM, and more) behind HTTP/REST and gRPC APIs. It supports cloud, data center, edge, and embedded deployments on NVIDIA GPUs, x86/ARM CPUs, and AWS Inferentia, with dynamic batching, ensembles, and streaming inference.

## Use cases
- serve pytorch models in production with a rest api
- deploy llm like llama2 or phi-3 with vllm or tensorrt-llm backend
- run batched gpu inference for many models from one server
- host onnx models for real-time inference at the edge
- serve models over grpc with dynamic batching and ensembles
- deploy models on nvidia gpus in docker containers
- stream audio or video inference requests to a model server

## When to choose
- you need production-grade, high-throughput model serving on NVIDIA GPUs
- you must serve models from multiple frameworks through a single API
- you want dynamic batching, model ensembles, and streaming out of the box
- you deploy across cloud, data center, and edge with the same tooling

## When to avoid
- you only need simple CPU-only inference for a single small model
- you want a lightweight embedded inference runtime rather than a server
- your stack is tightly tied to a non-NVIDIA accelerator without supported backends

## Facets
- artifact type: service
- maturity: stable
- function: llm-inference, machine-learning, deep-learning, http-server, rpc, gpu-computing, monitoring
- domain: machine-learning, deep-learning, large-language-models, cloud-computing, developer-tools, self-hosted
- platform: cloud, self-hosted, cross-platform
- tags: inference-server, model-serving, nvidia, tensorrt, pytorch, onnx, batching, grpc, edge-deployment, model-repository, linux, docker, gpu

## Member repositories
- triton-inference-server/server (main) score 98

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:10:44.752594+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T17:17:19.703074+00:00, confidence not recorded.
  - readme: https://github.com/triton-inference-server/server (fetched 2026-08-28T04:10:44.752594+00:00, sha e48d0e50b083)
  - homepage: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html (fetched 2026-08-29T08:16:13.741500+00:00, sha e844fdedbdc2)
  - site_page: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/contents.html (fetched 2026-08-29T08:16:13.744204+00:00, sha ff5abf6672e7)
  - site_page: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/introduction/index.html (fetched 2026-08-29T08:16:13.745861+00:00, sha e844fdedbdc2)
  - site_page: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/introduction/release_notes.html (fetched 2026-08-29T08:16:13.747509+00:00, sha e4821debe262)
  - site_page: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/introduction/compatibility.html (fetched 2026-08-29T08:16:13.749184+00:00, sha dcbe515801cd)
  - site_page: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/getting_started/quick_deployment.html (fetched 2026-08-29T08:16:13.751066+00:00, sha 27d18f71d91f)
  - site_page: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/getting_started/quickstart.html (fetched 2026-08-29T08:16:13.752496+00:00, sha 0c3e84f637c2)
  - site_page: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/getting_started/llm.html (fetched 2026-08-29T08:16:13.754308+00:00, sha 9cb4d4bcbc75)
  - site_page: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/tutorials/Popular_Models_Guide/Llama2/vllm_guide.html (fetched 2026-08-29T08:16:13.756969+00:00, sha 53cec4a5d0c3)
- Data as of 2026-08-30T08:39:29.467469+00:00.