microsoft/onnxruntime-genai

Health v2 · maintenance only

Adoption not part of the score

1142 stars · 371 forks observed · 2026-10-22

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

ONNX Runtime GenAI is a C++ library with Python, C#, C/C++, and Java APIs for running generative AI models (LLMs, Whisper, vision-language models) on device using ONNX Runtime. It implements the full generative loop including tokenization, logits processing, sampling, KV cache management, and constrained decoding for tool calling.

Use cases

• run llms locally on windows cpu or gpu
• on-device inference of phi-3 or llama models
• run whisper speech recognition with onnx runtime
• integrate local llm inference into a c# or c++ application
• run generative ai models on android devices
• quantized llm inference with cuda or directml acceleration

When to choose

• you need to run LLMs on-device with ONNX Runtime across Windows, Linux, Mac, or Android
• you want hardware acceleration via CPU, CUDA, DirectML, QNN, or OpenVINO without changing code
• you need multi-language APIs (Python, C#, C/C++, Java) for local generative AI inference

When to avoid

• you want server-side or cloud LLM inference with vLLM-style batching
• your model architecture is not in the supported matrix and you cannot export it to ONNX
• you need full multi-modal or speculative decoding support that is still on the roadmap

Facets

library · maturity active

llm-inference machine-learning sdk nlp large-language-models machine-learning artificial-intelligence developer-tools windows python cpp cross-platform onnx on-device-ai kv-cache tokenization sampling csharp java windows-ml foundry-local linux macos android gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="microsoft/onnxruntime-genai")