# microsoft/onnxruntime-genai

Generative AI extensions for onnxruntime

Repository: https://github.com/microsoft/onnxruntime-genai
Canonical: https://ross.abutalabs.com/products/onnxruntime-genai
Language: C++
License: MIT
License Family: permissive
Last push: 2026-09-03T00:34:41+00:00

## Health v2 (maintenance only)
Score: 93/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 100, release rhythm 96, longevity 73
- inputs: {"age_days": 1024, "days_push": 0, "days_rel": 26, "gap_med": 19.5, "n_releases_24m": 29}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1116, forks 350 (observed 2026-09-03T02:15:15.045345+00:00)

## What it is
ONNX Runtime GenAI is a C++ library with Python, C#, C/C++, and Java APIs for running generative AI models (LLMs, Whisper, vision-language models) on device using ONNX Runtime. It implements the full generative loop including tokenization, logits processing, sampling, KV cache management, and constrained decoding for tool calling.

## Use cases
- run llms locally on windows cpu or gpu
- on-device inference of phi-3 or llama models
- run whisper speech recognition with onnx runtime
- integrate local llm inference into a c# or c++ application
- run generative ai models on android devices
- quantized llm inference with cuda or directml acceleration

## When to choose
- you need to run LLMs on-device with ONNX Runtime across Windows, Linux, Mac, or Android
- you want hardware acceleration via CPU, CUDA, DirectML, QNN, or OpenVINO without changing code
- you need multi-language APIs (Python, C#, C/C++, Java) for local generative AI inference

## When to avoid
- you want server-side or cloud LLM inference with vLLM-style batching
- your model architecture is not in the supported matrix and you cannot export it to ONNX
- you need full multi-modal or speculative decoding support that is still on the roadmap

## Facets
- artifact type: library
- maturity: active
- function: llm-inference, machine-learning, sdk, nlp
- domain: large-language-models, machine-learning, artificial-intelligence, developer-tools
- platform: windows, python, cpp, cross-platform
- tags: onnx, on-device-ai, kv-cache, tokenization, sampling, csharp, java, windows-ml, foundry-local, linux, macos, android, gpu

## Member repositories
- microsoft/onnxruntime-genai (main) score 93

## Provenance
- Observed fields: from GitHub, fetched 2026-09-03T02:15:15.045345+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T06:43:36.622466+00:00, confidence not recorded.
  - readme: https://github.com/microsoft/onnxruntime-genai (fetched 2026-09-03T02:15:15.045345+00:00, sha 703b8c4f0f09)
- Data as of 2026-08-30T08:39:29.467469+00:00.