# ai-dynamo/dynamo

A Datacenter Scale Distributed Inference Serving Framework

Repository: https://github.com/ai-dynamo/dynamo
Canonical: https://ross.abutalabs.com/products/dynamo
Homepage: https://docs.nvidia.com/dynamo/latest
Language: Rust
License: NOASSERTION
License Family: other
Topics: diffusion, disaggregated-serving, kubernetes, llm-inference, omni, routing-engine, rust, sglang, tensorrt-llm, vllm
Last push: 2026-08-26T23:53:14+00:00

## Health v2 (maintenance only)
Score: 87/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 99, release rhythm 99, longevity 39
- inputs: {"age_days": 548, "days_push": 7, "days_rel": 11, "gap_med": 14.5, "n_releases_24m": 31}
- flags: no_license
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 7872, forks 1496 (observed 2026-08-28T04:10:08.323500+00:00)

## What it is
NVIDIA Dynamo is an open-source, datacenter-scale distributed inference serving framework that orchestrates inference engines like vLLM, SGLang, and TensorRT-LLM into coordinated multi-node systems. It provides disaggregated serving, KV-aware routing, multi-tier KV caching, and autoscaling, deployable on Kubernetes, Slurm, or locally.

## Use cases
- serve LLMs at datacenter scale across multiple nodes
- run disaggregated prefill and decode serving for lower latency
- route requests with KV-cache-aware routing to maximize cache reuse
- offload and manage KV cache across memory tiers
- autoscale inference workers on Kubernetes
- deploy models with vLLM, SGLang, or TensorRT-LLM backends
- serve multimodal and video generation workloads
- benchmark inference performance with AIPerf

## When to choose
- you need multi-node, datacenter-scale LLM inference with high throughput
- you want disaggregated prefill/decode serving with intelligent routing
- you deploy inference on Kubernetes or Slurm with NVIDIA, AMD, or Intel accelerators
- you want an orchestration layer on top of existing inference engines rather than replacing them

## When to avoid
- you just need to serve a single model on one GPU without distributed orchestration
- you need a simple inference server with minimal operational complexity
- your stack doesn't use supported engines like vLLM, SGLang, or TensorRT-LLM

## Facets
- artifact type: framework
- maturity: active
- function: llm-inference, routing, caching, deployment, microservices, api-gateway, monitoring, benchmarking
- domain: large-language-models, artificial-intelligence, gpu-computing, microservices, cloud-computing
- platform: python, rust, cli, self-hosted
- tags: inference-serving, disaggregated-serving, kv-cache, vllm, sglang, tensorrt-llm, nvidia, autoscaling, kv-aware-routing, prefill-decode-disaggregation, containers, devops, kubernetes, docker, linux, gpu

## Member repositories
- ai-dynamo/dynamo (main) score 87

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:10:08.323500+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T17:33:44.280711+00:00, confidence not recorded.
  - readme: https://github.com/ai-dynamo/dynamo (fetched 2026-08-28T04:10:08.323500+00:00, sha e8da334fcb86)
  - homepage: https://docs.nvidia.com/dynamo/latest (fetched 2026-08-29T08:30:17.395783+00:00, sha 23d31e034815)
  - site_page: https://docs.nvidia.com/dynamo/kubernetes/getting-started/introduction (fetched 2026-08-29T08:30:17.398449+00:00, sha df7a8511c8ad)
  - site_page: https://docs.nvidia.com/dynamo/cli/getting-started/introduction (fetched 2026-08-29T08:30:17.400258+00:00, sha bb981bcd1d9c)
  - site_page: https://www.nvidia.com/en-us/about-nvidia/privacy-policy (fetched 2026-08-29T08:30:17.401810+00:00, sha 5362c58d0750)
  - site_page: https://www.nvidia.com/en-us/about-nvidia/privacy-center (fetched 2026-08-29T08:30:17.404048+00:00, sha b098377da9cc)
  - site_page: https://www.nvidia.com/en-us/about-nvidia/terms-of-service (fetched 2026-08-29T08:30:17.405569+00:00, sha 85469b6ff0a1)
  - site_page: https://www.nvidia.com/en-us/about-nvidia/accessibility (fetched 2026-08-29T08:30:17.408162+00:00, sha 8f61ce4143ec)
  - site_page: https://www.nvidia.com/en-us/about-nvidia/company-policies (fetched 2026-08-29T08:30:17.410896+00:00, sha b169611fd7d6)
- Data as of 2026-08-30T08:39:29.467469+00:00.