triton-inference-server/server
The Triton Inference Server provides an optimized cloud and edge inferencing solution. observed · 2026-08-28
Health v2 · maintenance only
98/100
- Activity 99
- Release rhythm 95
- Longevity 100
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-03. Adoption (stars, forks) is never an input.
- gap_med: 28.0
- age_days: 2890
- days_rel: 35
- days_push: 7
- n_releases_24m: 23
Adoption not part of the score
10939 stars · 1830 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded
NVIDIA Triton Inference Server is an open-source inference serving software that deploys AI models from multiple frameworks (TensorRT, PyTorch, ONNX, OpenVINO, Python, vLLM, and more) behind HTTP/REST and gRPC APIs. It supports cloud, data center, edge, and embedded deployments on NVIDIA GPUs, x86/ARM CPUs, and AWS Inferentia, with dynamic batching, ensembles, and streaming inference.
Use cases
- serve pytorch models in production with a rest api
- deploy llm like llama2 or phi-3 with vllm or tensorrt-llm backend
- run batched gpu inference for many models from one server
- host onnx models for real-time inference at the edge
- serve models over grpc with dynamic batching and ensembles
- deploy models on nvidia gpus in docker containers
- stream audio or video inference requests to a model server
When to choose
- you need production-grade, high-throughput model serving on NVIDIA GPUs
- you must serve models from multiple frameworks through a single API
- you want dynamic batching, model ensembles, and streaming out of the box
- you deploy across cloud, data center, and edge with the same tooling
When to avoid
- you only need simple CPU-only inference for a single small model
- you want a lightweight embedded inference runtime rather than a server
- your stack is tightly tied to a non-NVIDIA accelerator without supported backends
Facets
service · maturity stable
llm-inference machine-learning deep-learning http-server rpc gpu-computing monitoring machine-learning deep-learning large-language-models cloud-computing developer-tools self-hosted cloud self-hosted cross-platform inference-server model-serving nvidia tensorrt pytorch onnx batching grpc edge-deployment model-repository linux docker gpu
10 sources
- readme: https://github.com/triton-inference-server/server · fetched 2026-08-28 · e48d0e50b083
- homepage: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html · fetched 2026-08-29 · e844fdedbdc2
- site_page: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/contents.html · fetched 2026-08-29 · ff5abf6672e7
- site_page: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/introduction/index.html · fetched 2026-08-29 · e844fdedbdc2
- site_page: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/introduction/release_notes.html · fetched 2026-08-29 · e4821debe262
- site_page: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/introduction/compatibility.html · fetched 2026-08-29 · dcbe515801cd
- site_page: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/getting_started/quick_deployment.html · fetched 2026-08-29 · 27d18f71d91f
- site_page: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/getting_started/quickstart.html · fetched 2026-08-29 · 0c3e84f637c2
- site_page: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/getting_started/llm.html · fetched 2026-08-29 · 9cb4d4bcbc75
- site_page: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/tutorials/Popular_Models_Guide/Llama2/vllm_guide.html · fetched 2026-08-29 · 53cec4a5d0c3
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| triton-inference-server/server | main | 98 |
For agents
markdown · JSON · MCP: product_card(name="triton-inference-server/server")
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem