Open-source llm-inference software

3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.

3358 products, primary matches first, then adoption-weighted; health v2 shown.

Product	Health v2	Stars	Maturity
charmbracelet/mods Mods is a Go CLI tool that brings LLM-based AI to the command line, designed to work in shell pipelines by ingesting command output and for…	10	4522	maintenance
kellyvv/PhoneClaw PhoneClaw is a mobile-native local AI agent framework that turns phones into on-device agent runtimes, running Gemma models via LiteRT and …	71	1279	active
DreamLM/Dream Dream 7B is an open diffusion large language model (dLLM) with base and instruct checkpoints, plus inference and training code built on Hug…	40	1279	active
ABexit/ASR-LLM-TTS An open-source Python speech interaction pipeline that chains SenseVoice ASR, Qwen2.5 LLMs, and TTS engines (CosyVoice, Edge-TTS, pyttsx3) …	57	1278	active
stair-lab/kg-gen kg-gen is a Python library that extracts knowledge graphs from arbitrary plain text or conversation messages using LLMs, with model routing…	57	1278	active
hyperonym/basaran Basaran is an open-source alternative to the OpenAI text completion API that serves Hugging Face Transformers-based text generation models …	10	1278	active
fxy2311-youyou/expression-trainer A local desktop app (Electron) that trains spoken expression skills by combining fully offline real-time speech recognition (Sherpa-ONNX) w…	52	1276	active
FoloUp/FoloUp FoloUp is an open-source, self-hostable web application that conducts AI-powered voice interviews with job candidates. It generates tailore…	55	1275	active
ai-genie/chatgpt-vscode Genie AI is a Visual Studio Code extension that integrates OpenAI's ChatGPT models (GPT-4o, GPT-4, GPT-3.5, o1) as an AI pair programmer. I…	21	1275	active
ModelCloud/GPTQModel GPTQModel is a production-ready Python toolkit for quantizing (compressing) large language models using GPTQ, AWQ, and related methods, wit…	92	1274	active
ipa-lab/hackingBuddyGPT HackingBuddyGPT is a Python framework that helps ethical hackers and security researchers use LLMs and LLM-based autonomous agents for pene…	66	1274	active
okisdev/ChatChat Chat Chat is a self-hostable web platform that unifies chat and search interfaces to multiple AI providers such as OpenAI, Anthropic, Coher…	65	1271	active
Xerxes-2/clewdr ClewdR is a high-performance Rust proxy for Claude.ai and Claude Code that exposes native Claude and OpenAI-compatible API endpoints from a…	78	1270	active
ICT-FinD-Lab/alphagen AlphaGen is a Python research library that automatically generates formulaic alpha (predictive) stock factors using reinforcement learning,…	67	1268	active
yym68686/uni-api uni-api is a self-hosted Python service that unifies management of multiple LLM provider APIs behind a single OpenAI-compatible endpoint. I…	84	1266	active
TencentCloudADP/youtu-graphrag Youtu-GraphRAG is a Python framework for graph-based retrieval-augmented generation that unifies graph schema construction, community detec…	52	1266	active
Storia-AI/sage Sage is an open-source tool that lets developers chat with any codebase to quickly understand how it works, similar to a self-hosted GitHub…	10	1265	active
SimonSchubert/Kai Kai 9000 is an open-source, cross-platform AI assistant built with Kotlin Multiplatform that runs on Android, iOS, Windows, Mac, Linux, and…	81	1264	active
techjarves/Uncensored-Local-AI-Multiplatform A cross-platform Flutter application that runs uncensored open-source GGUF language models entirely on-device for Android, iOS, and desktop…	62	1263	active
HelgeSverre/ollama-gui Ollama GUI is a web-based chat interface for interacting with local LLMs served through the Ollama API. It offers local chat history via In…	95	1258	active
thu-ml/SpargeAttn SpargeAttn is the official implementation of a training-free sparse and quantized attention method that accelerates inference for language,…	48	1257	active
hkjarral/AVA-AI-Voice-Agent-for-Asterisk An open-source AI voice agent that integrates with Asterisk/FreePBX phone systems via Audiosocket/RTP, built in Python with a modular pipel…	85	1256	active
MoonshotAI/FlashKDA FlashKDA is a set of high-performance CUDA kernels (built on CUTLASS) implementing Kimi Delta Attention, a linear attention mechanism, for …	56	1256	active
TIGER-AI-Lab/OpenResearcher OpenResearcher is a fully open-source pipeline for synthesizing long-horizon deep research trajectories using LLM agents with retrieval and…	51	1254	active
facebookresearch/llm-transparency-tool An interactive toolkit from Meta Research for analyzing the internal workings of Transformer-based language models. It visualizes contribut…	10	1253	active
RUC-NLPIR/Search-o1 Search-o1 is a research framework that enhances large reasoning models (like QwQ and R1) with agentic search and retrieval-augmented genera…	41	1247	active
openai/codex-action A GitHub Action that runs OpenAI's Codex CLI inside GitHub Actions workflows, installing the CLI and routing API calls through a secure pro…	60	1246	active
MaoXiaoYuZ/Long-Novel-GPT Long-Novel-GPT is a self-hosted web application that uses LLMs and RAG to generate and revise long-form novels through a top-down outline-c…	46	1246	active
xllm-go/bypass A Go server that reverse-engineers the chat interfaces of multiple AI providers (Coze, DeepSeek, Cursor, Windsurf, Grok, Bing Copilot, You,…	56	1245	active
mrwadams/attackgen AttackGen is a Streamlit-based cybersecurity tool that uses large language models to generate tailored incident response testing scenarios …	91	1244	active
sums001/Windows-Copilot-API A Python library and local server that reverse engineers Microsoft Copilot's free web chat into an OpenAI-compatible REST API, giving acces…	50	1244	active
HJYao00/Mulberry Mulberry is a research implementation of an o1-like multimodal large language model (MLLM) that performs step-by-step reasoning and reflect…	46	1244	active
Ksuriuri/index-tts-vllm A reimplementation of IndexTTS's GPT model inference using vLLM, providing significantly faster text-to-speech generation with a web UI and…	53	1243	active
AWorld AWorld is an open-source agent framework and runtime that orchestrates tools, memory, context, and execution for building autonomous AI age…	76	1242	active
goodroot/hyprwhspr hyprwhspr is a native Linux system-wide speech-to-text dictation application supporting local models (Whisper, Parakeet, Cohere) with optio…	81	1240	active
mrousavy/react-native-fast-tflite A high-performance TensorFlow Lite library for React Native built on Nitro Modules, using the low-level C/C++ TFLite core API with zero-cop…	81	1240	active
0xacx/chatGPT-shell-cli A lightweight shell script that lets you chat with OpenAI's ChatGPT models and generate DALL-E images directly from the terminal, requiring…	32	1240	active
Shopify/roast Roast is a Ruby-based domain-specific language for building structured AI workflows from composable building blocks called cogs. It orchest…	80	1239	active
pytorch/serve TorchServe is a flexible, production-ready model server for serving, optimizing, and scaling PyTorch models over HTTP with support for CPU,…	10	4344	maintenance
Michaelliv/pi-dynamic-workflows A Pi extension that adds a `workflow` tool enabling the model to write JavaScript scripts that fan out work across isolated subagents and s…	64	1236	active
laravel/ai The Laravel AI SDK is a PHP package offering a unified, expressive API for interacting with AI providers such as OpenAI, Anthropic, and Gem…	84	1234	active
modu-ai/moai-adk MoAI-ADK is an agentic development harness for Claude Code that enforces SPEC-driven plan/run/sync workflows, TRUST 5 quality gates, and mu…	78	1234	active
qualcomm/ai-hub-models Qualcomm AI Hub Models is a curated collection of 300+ state-of-the-art machine learning models (vision, audio, speech, generative AI) pre-…	90	1230	active
OpenTSLM/OpenTSLM OpenTSLM is a family of Time-Series Language Models that integrate time series as a native modality into pretrained LLMs (Llama, Gemma), en…	58	1230	active
wanxingai/LightAgent LightAgent is an ultra-lightweight open-source Python framework for building LLM-powered agents with tools, persistent memory, guardrails, …	87	1229	active
microsoft/MInference MInference is a Microsoft library that accelerates long-context LLM inference using dynamic sparse attention, reducing pre-fill latency by …	60	1229	active
pireel/pireel Pireel Studio is an open-source, browser-based AI video editor for talking heads, shorts, demos, and montages, with canvas/timeline editing…	56	1228	active
langchain-ai/openevals OpenEvals is a library of readymade evaluators for testing LLM applications, including LLM-as-judge evaluators and reference-based/referenc…	89	1227	active
MoonshotAI/Kimi-VL Kimi-VL is an open-source Mixture-of-Experts vision-language model (VLM) with a 2.8B activated parameter language decoder, offering multimo…	31	1227	active
run-house/kubetorch Kubetorch is a Python library that lets you distribute and run ML workloads (training, inference, data processing) on Kubernetes directly f…	76	1225	active
Opencode-DCP/opencode-dynamic-context-pruning A plugin for OpenCode that dynamically prunes and compresses conversation context to reduce token usage when working with LLMs. It replaces…	77	4294	maintenance
Truss Truss is a Python CLI and packaging framework for deploying and serving AI/ML models in production, primarily on the Baseten platform. It h…	96	1223	active
FlowDown FlowDown is a native AI chat client for iPhone, iPad, and Mac built with Swift and UIKit. It connects to OpenAI-compatible APIs, self-hoste…	85	1223	active
allwefantasy/auto-coder Auto-Coder is an open-source AI coding agent and CLI tool (powered by Byzer-LLM) that provides chat, one-shot command, server, and RAG mode…	66	1221	active
vincanger/coverlettergpt A full-stack web application that generates tailored cover letters from a job description and the user's resume using OpenAI's GPT models. …	40	1220	active
moltlaunch/cashclaw CashClaw is an autonomous AI agent that connects to the Moltlaunch onchain marketplace, where it evaluates tasks, quotes prices, executes w…	44	1219	active
AllAboutAI-YT/easy-local-rag A Python application providing 100% local retrieval-augmented generation (RAG) using Ollama for LLM inference and embeddings. It supports i…	16	1219	active
tonyqinatcmu/SlideBot-AI SlideBot AI is an AI-powered presentation generator that turns a topic, outline, or uploaded materials (documents, spreadsheets, meeting re…	42	1216	active
RhythmicWave/NovelForge NovelForge is an AI-assisted long-form novel writing application built in Python, using a card-based creation model with JSON Schema-driven…	83	1215	active
Symfony AI Symfony AI is a set of PHP components and Symfony bundles that integrate AI capabilities into PHP applications, including a unified interfa…	85	1214	active
Zefan-Cai/R-KV R-KV is a training-free, redundancy-aware KV cache compression method for reasoning LLMs, discarding repetitive tokens on-the-fly during de…	64	1214	active
anthropics/anthropic-sdk-go The official Go SDK for accessing Anthropic's Claude language model APIs. It provides typed clients for messages, streaming, and other Clau…	85	1212	active
Morgensonne/EditDeck EditDeck is an end-to-end pipeline that turns plain-language requirements into structured slide outlines, rendered slide images, standard P…	47	1212	active
NVIDIA/kvpress kvpress is a Python library from NVIDIA that implements multiple KV cache compression methods and benchmarks for long-context LLM inference…	86	1211	active
xenodium/chatgpt-shell An Emacs package providing a multi-LLM shell and chat interface supporting ChatGPT, Claude, DeepSeek, Gemini, Kagi, Ollama, and Perplexity,…	74	1210	active
modal-labs/quillman QuiLLMan is a voice chat application built on Kyutai's Moshi speech-to-speech language model, deployed serverlessly on Modal with a FastAPI…	65	1210	active
jieyefriic/rp-engine A YAML-native agent workflow execution engine written in Rust that parses declarative workflow files describing nodes, edges, prompts, data…	61	1209	active
Elixir LangChain An Elixir framework for building LLM-powered applications, providing abstractions and integrations for chat models like OpenAI, Anthropic C…	95	1208	active
amruth-sn/kong Kong is an LLM-orchestrated reverse engineering CLI that plugs AI into Ghidra's analysis engine to recover function names, types, and struc…	57	1208	active
cvs-health/uqlm UQLM is a Python library for detecting LLM hallucinations using uncertainty quantification techniques. It provides five categories of score…	86	1207	active
clacky-ai/openclacky OpenClacky is an open-source (MIT) AI agent platform written in Ruby that runs locally, offering a CLI and Web UI for completing tasks via …	80	1207	active
microsoft/RecAI RecAI is a Microsoft research project exploring ways to integrate large language models into recommender systems (LLM4Rec). It includes a c…	54	1205	active
googleapis/go-genai The official Google Gen AI SDK for Go, providing a client interface to Google's generative models such as Gemini via the Gemini Developer A…	85	1204	active
nishuzumi/gemini-teacher A Python CLI application that acts as an English speaking practice assistant powered by Google Gemini. It listens to your speech via microp…	65	1203	active
CreminiAI/skillpack SkillPack is an open-source CLI and runtime for packaging AI skills (prompts, scripts, tools) into distributable zip bundles that run as lo…	58	1203	active
Artelnics/opennn OpenNN is an open-source C++ library for building, training, and deploying neural networks for advanced analytics. It is dependency-free, o…	94	1202	active
openvinotoolkit/nncf NNCF is Intel's Neural Network Compression Framework, a Python library providing post-training and training-time compression algorithms (qu…	92	1202	active
heyhuynhgiabuu/proxypal ProxyPal is a native desktop app (Tauri-based) that wraps CLIProxyAPI to expose AI subscriptions like Claude, ChatGPT, Gemini, and GitHub C…	76	1202	active
sozercan/kubectl-ai A kubectl plugin written in Go that generates and applies Kubernetes manifests from natural-language prompts using OpenAI GPT or OpenAI API…	21	1196	active
Zen4-bit/Proxima Proxima is a local-first Electron application that runs a multi-AI Model Context Protocol (MCP) server, routing ChatGPT, Claude, Gemini, an…	69	1195	active
huggingface/llm.nvim llm.nvim is a Neovim plugin that brings LLM-powered features like ghost-text code completion to the editor, using llm-ls as its backend. It…	73	1194	active
mckaywrigley/ai-code-translator A web application that uses AI (LLMs) to translate source code from one programming language to another. It is a Next.js/TypeScript app tha…	31	4165	maintenance
Soul-AILab/SoulX-LiveAct SoulX-LiveAct is the official inference code for a real-time human animation framework that generates lifelike, audio/multimodal-controlled…	51	1193	active
lmstudio-ai/mlx-engine mlx-engine is the Apple MLX-based LLM inference engine that powers LM Studio on Mac, built on mlx-lm with support for vision models via mlx…	68	1192	active
dulikaifazr/Cougar-CLI Cougar CLI is an AI programming agent that runs in the terminal, letting developers interact with LLMs to write and manage code from the co…	42	1192	active
GAIR-NLP/ASI-Arch A multi-agent framework that lets an LLM autonomously conduct end-to-end research on neural network architecture discovery, specifically li…	40	1192	active
linux-do/override A Go-based local proxy service that overrides GitHub Copilot's API endpoints in VSCode and JetBrains IDEs, redirecting code completion and …	15	1191	active
zai-org/VisualGLM-6B VisualGLM-6B is an open-source multimodal conversational language model supporting images, Chinese, and English, built on ChatGLM-6B with a…	30	4154	maintenance
polyuiislab/infiAgent infiAgent (MLA) is a Python agent framework for building domain-specific AI agents that run for days on complex tasks without context degra…	66	1190	active
jncraton/languagemodels A Python library providing simple building blocks for running large language models locally with as little as 512MB of RAM. It offers instr…	57	1189	active
fmaclen/hollama Hollama is a minimal LLM chat application that runs entirely in the browser, connecting to Ollama and OpenAI servers with all data stored l…	93	1188	active
Cloudflare AI A monorepo of TypeScript packages and examples for building AI-powered applications on Cloudflare. It provides Vercel AI SDK and TanStack A…	79	1188	active
Agent Landing Zone GPT-RAG is an Azure solution accelerator providing architecture templates and deployment assets for enterprise-scale Retrieval-Augmented Ge…	92	1187	active
ALwrity/ALwrity ALwrity is an open-source, AI-first digital marketing platform built in Python that combines content strategy planning, multimodal AI conte…	77	1186	active
madox2/vim-ai A Vim/Neovim plugin that adds AI capabilities such as code generation, text editing, and interactive ChatGPT conversations directly in the …	76	1186	active
SamuelZ12/longcut LongCut is a Next.js web application that turns long-form YouTube videos into a structured learning workspace with AI-generated highlight r…	61	1186	active
MediaBrain-SJTU/MING MING (明医) is a Chinese medical consultation large language model fine-tuned on medical instruction data, with variants built on bloomz-7b a…	37	1186	active
DAMO-NLP-SG/VideoLLaMA3 VideoLLaMA 3 is a frontier multimodal foundation model for image and video understanding, released with checkpoints, inference code, and de…	34	1186	active
Foundation Models Framework A native iOS and macOS workbench app for learning, testing, and evaluating Apple's Foundation Models framework. It provides editable recipe…	78	1185	active
auto-novel/auto-novel AutoNovel is a website application that automatically machine-translates light novels (web novels, bunko novels, and local files) using LLM…	76	1185	active