Open-source llm-inference software
3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
3358 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| huggingface/swift-coreml-diffusers A native SwiftUI application demonstrating how to run Stable Diffusion text-to-image generation on-device using Apple's Core ML Stable Diff… | 10 | 2755 | active |
| thewh1teagle/kokoro-onnx A Python library that runs the Kokoro text-to-speech model via ONNX Runtime, supporting CPU and GPU inference. It provides multi-language T… | 65 | 2750 | active |
| iamsrikanthnani/pluely Pluely is a privacy-first desktop AI assistant that runs as an invisible always-on-top overlay, providing live meeting transcription, on-sc… | 82 | 2746 | active |
| codeany-ai/open-agent-sdk-typescript An open-source TypeScript Agent SDK that runs the full agent loop in-process without CLI or subprocess dependencies, as an alternative to c… | 54 | 2746 | active |
| lightseekorg/tokenspeed TokenSpeed is a high-performance LLM inference engine designed for agentic workloads, aiming for TensorRT-LLM-level performance with vLLM-l… | 78 | 2743 | active |
| Zero6992/chatGPT-discord-bot A self-hosted Discord bot that integrates ChatGPT-style AI chat into Discord servers, supporting multiple AI providers including a free pro… | 84 | 2735 | active |
| 1rgs/nanocode A minimal Claude Code alternative implemented as a single ~250-line Python file with zero dependencies. It runs an agentic loop with file a… | 41 | 2735 | active |
| BigDawnGhost/wenyi Wenyi is a Python CLI tool that translates EPUB books into Chinese using LLMs, with whole-book context analysis, real-time glossary extract… | 80 | 2733 | active |
| xdit-project/xDiT xDiT is a scalable inference engine for Diffusion Transformers (DiTs) that enables parallel deployment across multiple GPUs and machines. I… | 91 | 2732 | active |
| Haleclipse/CodexDesktop-Rebuild A cross-platform Electron rebuild of the OpenAI Codex Desktop app, distributed for macOS, Windows, and Linux. It packages the Codex coding … | 76 | 2725 | active |
| intel/neural-compressor Intel Neural Compressor is an open-source Python library providing state-of-the-art low-bit quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4),… | 95 | 2723 | active |
| HuaYaoAI/FinGenius FinGenius is an open-source AI financial analysis platform for China's A-share stock market, built on a multi-agent architecture where spec… | 32 | 2723 | active |
| AmberSahdev/Open-Interface Open Interface is a cross-platform desktop application that lets users control their computer using natural language requests processed by … | 62 | 2721 | active |
| black-forest-labs/flux2 Official inference repository for Black Forest Labs' FLUX.2 family of open-weight image generation and editing models. It provides minimal … | 45 | 2720 | active |
| getumbrel/llama-gpt LlamaGPT is a self-hosted, offline ChatGPT-like chatbot powered by Llama 2 and Code Llama models via llama.cpp, with all data staying on th… | 29 | 10926 | maintenance |
| KimMeen/Time-LLM Time-LLM is the official PyTorch implementation of an ICLR 2024 paper that reprograms frozen large language models (Llama, GPT-2, BERT) for… | 44 | 2707 | active |
| spring-ai-alibaba/DataAgent An enterprise-grade AI data analyst agent built on Spring AI Alibaba Graph that converts natural language to SQL, runs Python deep analysis… | 79 | 2702 | active |
| dreamzero0/dreamzero DreamZero is NVIDIA's World Action Model (WAM) that jointly predicts future video and actions from a pretrained video diffusion backbone, e… | 48 | 2700 | active |
| ZHZisZZ/dllm dLLM is a Python library that unifies training, inference, and evaluation of diffusion language models such as LLaDA and Dream. It builds o… | 56 | 2699 | active |
| TavernAI TavernAI is an atmospheric frontend application for roleplay chat and story writing with AI language models, connecting to backends like Ko… | 58 | 2698 | active |
| huggingface/text-generation-inference Text Generation Inference (TGI) is a Rust, Python and gRPC toolkit for deploying and serving large language models with high performance, p… | 10 | 10877 | maintenance |
| Mobile-Artificial-Intelligence/maid Maid is a free, open-source Android chat application (built with React Native) for talking to large language models. It runs GGUF models fu… | 81 | 2695 | active |
| openai/openai-dotnet The official .NET library for accessing the OpenAI REST API, generated from the OpenAPI specification in collaboration with Microsoft. It p… | 85 | 2694 | active |
| itayinbarr/little-coder little-coder is a terminal coding agent built on the pi agent loop, tuned for small local language models like Qwen via Ollama or llama.cpp… | 79 | 2691 | active |
| wassim249/fastapi-langgraph-agent-production-ready-template A production-ready FastAPI template for building AI agent backends with LangGraph integration. It provides stateful conversations, long-ter… | 63 | 2687 | active |
| nicepkg/aide Aide is a VSCode extension that brings AI-assisted coding features such as one-click code commenting, language/framework conversion, UI scr… | 20 | 2686 | active |
| bestruirui/octopus Octopus is a self-hosted LLM API aggregation gateway that unifies multiple provider channels behind a single endpoint, converting between O… | 79 | 2682 | active |
| huangserva/3DCellForge A React + Three.js web application for AI-powered interactive 3D model generation, inspection, and presentation. It turns uploaded referenc… | 55 | 2678 | active |
| abi/secret-llama Secret Llama is a fully private LLM chatbot that runs entirely in the browser with no server or installation required, powered by the WebLL… | 25 | 2675 | active |
| LigphiDonk/academic-figure-generator A self-hosted AI-powered platform that generates high-quality academic paper figures: users upload a paper (PDF/DOCX/TXT), Claude analyzes … | 56 | 2674 | active |
| om-ai-lab/OmAgent OmAgent is a Python library for building multimodal language agents, wrapping worker orchestration, task queues, and graph-based workflow o… | 28 | 2667 | active |
| kingjulio8238/Memary Memary is an open-source memory layer for autonomous AI agents that emulates human memory using knowledge graphs. It is a Python library (p… | 16 | 2661 | active |
| NeMo Agent Toolkit NVIDIA NeMo Agent Toolkit is an open-source Python library for connecting, orchestrating, and optimizing teams of AI agents across any agen… | 85 | 2660 | active |
| open-gigaai/giga-brain-0 GigaBrain-0/0.7 is an open-source vision-language-action (VLA) model family for generalist embodied agents, powered by world models and a t… | 60 | 2659 | active |
| Narrator AI A machine-readable SKILL.md skill package that teaches AI agents (Claude Code, OpenClaw, Windsurf, Cursor, etc.) how to drive the narrator-… | 53 | 2659 | active |
| Const-me/Whisper A Windows port of whisper.cpp that runs OpenAI's Whisper speech recognition model on the GPU via DirectCompute (Direct3D 11 compute shaders… | 66 | 10678 | maintenance |
| AnswerDotAI/llms-txt A proposal and Python tooling for the /llms.txt standard, a markdown file placed at a website root to give language models and agents conci… | 79 | 2650 | active |
| mito-ds/mito Mito is a set of JupyterLab extensions that combine an AI chat copilot, smart debugging, and an interactive spreadsheet interface that gene… | 73 | 2648 | active |
| aardio/ImTip ImTip is a lightweight (under 1 MB) Windows desktop assistant that shows real-time input method and keyboard state indicators at the text c… | 76 | 2646 | active |
| NVlabs/LongLive LongLive is an NVIDIA research framework providing parallel training and inference infrastructure for real-time long video generation, usin… | 60 | 2639 | active |
| run-llama/sec-insights SEC Insights is a full-stack reference application built by LlamaIndex that uses RAG to answer questions about SEC 10-K and 10-Q financial … | 29 | 2610 | active |
| Zafer-Liu/Data-Analysis-Agent An LLM-powered conversational data analysis agent that lets users connect data sources (Excel/CSV files, databases, Feishu tables) and ask … | 79 | 2608 | active |
| google-coral/coralnpu Coral NPU is an open-source neural processing unit (NPU) hardware IP core from Google Research, built on the 32-bit RISC-V ISA with matrix,… | 72 | 2604 | active |
| kayba-ai/agentic-context-engine Agentic Context Engine (ACE) is a Python library that lets LLM-based agents learn from experience by maintaining and evolving an evolving c… | 82 | 2602 | active |
| Memento-Teams/Memento Memento is a Python framework for building LLM agents that continually improve from experience via memory-based case-based reasoning, witho… | 36 | 2601 | active |
| CodeGeeX CodeGeeX is a family of open multilingual code generation large language models (13B and successors CodeGeeX2/CodeGeeX4) pre-trained on 20+… | 24 | 2597 | active |
| yazinsai/OpenOats OpenOats is a macOS meeting assistant that transcribes both sides of a call in real time using on-device speech recognition and surfaces re… | 76 | 2595 | active |
| BAAI-Agents/Cradle Cradle is a Python framework for General Computer Control (GCC), enabling foundation agents to perform complex computer tasks using screens… | 26 | 2594 | active |
| microsoft/foundry-local Foundry Local is Microsoft's end-to-end local AI runtime and SDK suite (C#, JavaScript, Python, Rust) for running optimized models entirely… | 87 | 2592 | active |
| ChatTTS_colab A one-click deployment wrapper around ChatTTS providing a Gradio web UI for text-to-speech, runnable in Google Colab or via an offline Wind… | 50 | 2592 | active |
| ahmedeltaher/Android-MVVM-Architecture-Android-Voice-AI-SDK A reusable Android library (Kotlin, MVVM) that provides a full voice-driven AI conversation pipeline: microphone capture with VAD, speech-t… | 67 | 2582 | active |
| Griptape Griptape is a modular Python framework for building generative AI applications, providing abstractions for agents, pipelines, workflows, RA… | 92 | 2581 | active |
| Sourcegraph Sourcegraph is a code intelligence platform providing universal code search, code navigation, batch changes, and the Cody AI coding assista… | 10 | 10299 | maintenance |
| Tencent-Hunyuan/HY-Motion-1.0 HY-Motion 1.0 is a series of text-to-3D human motion generation models from Tencent Hunyuan, based on Diffusion Transformer and Flow Matchi… | 55 | 2574 | active |
| YeQing17-2026/OmniAgent OmniAgent is an open-source Python agent framework that self-evolves across skills, context, memory, and its underlying model during intera… | 54 | 2570 | active |
| pykeio/ort ort is a safe, fast Rust binding for Microsoft's ONNX Runtime, providing hardware-accelerated machine learning inference and training for O… | 91 | 2569 | active |
| mirage-project/mirage Mirage Persistent Kernel (MPK) is a compiler and runtime that transforms multi-GPU LLM inference into a single fused megakernel, reducing i… | 82 | 2569 | active |
| huggingface/lighteval Lighteval is Hugging Face's all-in-one toolkit for evaluating large language models across multiple inference backends (Transformers, vLLM,… | 72 | 2563 | active |
| fogsightai/fogsight Fogsight is an open-source AI agent and animation engine powered by large language models that turns abstract concepts or words into polish… | 63 | 2561 | active |
| Hugging Face JavaScript SDK A collection of TypeScript JavaScript libraries for interacting with the Hugging Face ecosystem, including Hub repository management, unifi… | 95 | 2560 | active |
| DeepDetect DeepDetect is an open-source deep learning runtime, CLI, and REST server written in C++ for training and inference across images, text, tab… | 95 | 2551 | active |
| alexgreensh/token-optimizer A Claude Code plugin/skill that finds and removes wasted 'ghost tokens' in agent context windows, helping sessions survive compaction witho… | 78 | 2544 | active |
| SafeAILab/EAGLE EAGLE is the official implementation of the EAGLE family of speculative decoding algorithms (EAGLE-1/2/3) for accelerating LLM text generat… | 54 | 2542 | active |
| Perplexity API Platform The official Model Context Protocol (MCP) server for the Perplexity API Platform, exposing real-time web search, reasoning, and research to… | 62 | 2541 | active |
| aingdesk/AingDesk AingDesk is an open-source, MIT-licensed AI assistant application for running local AI models (e.g., via Ollama) and third-party model APIs… | 57 | 2539 | active |
| X-PLUG/mPLUG-Owl mPLUG-Owl is a family of open-source multimodal large language models that combine visual encoders with LLMs (built on LLaMA) for image and… | 33 | 2539 | active |
| nikmcfly/MiroFish-Offline MiroFish-Offline is a fully local, English-language fork of MiroFish, a multi-agent swarm intelligence engine that simulates public opinion… | 44 | 2538 | active |
| VITA-MLLM/VITA VITA is an open-source interactive omni multimodal large language model (VITA-1.5) that supports real-time vision and speech interaction, s… | 26 | 2536 | active |
| harry0703/AudioNotes AudioNotes is a locally-run web application that transcribes audio and video files into text and organizes them into structured Markdown no… | 64 | 2535 | active |
| Lamini The official Python client and SDK for Lamini's hosted API for building generative AI applications. It lets developers call Lamini's LLM in… | 34 | 2533 | active |
| alecm20/story-flicks Story-Flicks is a Python + FastAPI application with a React frontend that generates high-definition story short videos from a user-provided… | 22 | 2531 | active |
| cixingguangming55555/wechat-bot A free toolset (funtool) providing proxy executables that let the Claude Code CLI use free NVIDIA-hosted large models such as Kimi K2.5, GL… | 58 | 2521 | active |
| hydropix/TranslateBooksWithLLMs A desktop application that translates full-length books, subtitles, and documents using local or cloud LLMs (Ollama, OpenAI-compatible APIs… | 82 | 2517 | active |
| youssofal/MTPLX MTPLX is a native macOS app and CLI that runs local LLMs on Apple Silicon roughly twice as fast using native multi-token prediction (MTP) s… | 80 | 2511 | active |
| InternLM/HuixiangDou HuixiangDou is an LLM-based professional knowledge assistant designed for group chat scenarios, using a three-stage pipeline of preprocess,… | 41 | 2501 | active |
| KsanaDock/Microverse Microverse is a god-simulation sandbox game built on Godot 4 that implements a multi-agent AI social simulation. AI characters have indepen… | 48 | 2500 | active |
| janhq/ichigo Ichigo is a Python speech package for developers offering local realtime voice AI capabilities, including a compact 22M-parameter speech to… | 47 | 2495 | active |
| K-Dense-AI/claude-scientific-writer An AI-powered deep research and scientific writing tool that generates publication-ready papers, reports, posters, grant proposals, and lit… | 83 | 2493 | active |
| neavo/LinguaGacha LinguaGacha is an AI-powered desktop text translator that batch-translates novels, game scripts, subtitles, and ebooks using LLM APIs (Open… | 81 | 2488 | active |
| Nano-Collective/nanocoder Nanocoder is an open-source, terminal-based AI coding agent built by a community collective. It runs agentic coding workflows against any m… | 77 | 2488 | active |
| 24mlight/A_Share_investment_Agent A Python-based multi-agent AI investment analysis system for China's A-share market that uses LLMs (Gemini/OpenAI-compatible) with bull/bea… | 56 | 2482 | active |
| google/XNNPACK XNNPACK is a highly optimized C library of floating-point neural network inference operators for ARM, x86, WebAssembly, and RISC-V platform… | 77 | 2481 | active |
| darrenburns/elia Elia is a keyboard-centric terminal user interface (TUI) application for chatting with large language models like ChatGPT, Claude, Gemini, … | 21 | 2481 | active |
| Live-GalGame/LiveGalGame LiveGalGame is a playful app that overlays a visual-novel (galgame) interface onto real-life conversations, providing real-time speech-to-t… | 52 | 2477 | active |
| microsoft/PIKE-RAG PIKE-RAG is a Microsoft framework for building Retrieval-Augmented Generation systems that extract, understand, and apply specialized domai… | 10 | 2475 | active |
| roboflow/inference Roboflow Inference is a Python library and self-hostable inference server for deploying computer vision models on any computer or edge devi… | 91 | 2474 | active |
| kaqijiang/Auto-GPT-ZH A Chinese-language community fork and translation of AutoGPT, the autonomous AI agent platform, kept in sync with the upstream stable branc… | 34 | 2464 | active |
| inference-labs-inc/subnet-2 Subnet 2 on the Bittensor network implements a Proof-of-Inference system that uses zero-knowledge machine learning (zk-ML) to cryptographic… | 86 | 2462 | active |
| theJayTea/WritingTools A system-wide AI writing assistant for Windows, Linux, and macOS that proofreads, rewrites, and summarizes text in any application via a ho… | 77 | 2460 | active |
| xusenlinzy/api-for-open-llm A Python server that exposes open-source large language models (LLaMA, ChatGLM, Qwen, Baichuan, Falcon, etc.) behind an OpenAI-compatible A… | 21 | 2459 | active |
| sqliteai/warp WARP is a dependency-free, embeddable C inference engine that runs trillion-parameter mixture-of-experts models like Kimi K3 by streaming a… | 77 | 2457 | active |
| leilei926524-tech/anti-distill A tool that sanitizes AI Skill files employees are asked to write, producing a professional-looking but hollowed-out version for submission… | 52 | 2455 | active |
| volcengine/ai-app-lab AI App Lab from Volcano Engine (Volcengine) provides Arkitect, a high-code Python SDK for building LLM applications, plus Demohouse, a coll… | 78 | 2451 | active |
| ggozad/oterm oterm is a terminal-based chat client for LLMs, supporting Ollama, OpenAI, Anthropic, and any pydantic-ai-supported provider. It offers a r… | 95 | 2448 | active |
| 1sdv/TripStar TripStar is an AI-powered travel planning application built on the HelloAgents multi-agent framework, using LLMs to generate personalized t… | 58 | 2446 | active |
| FullAgent/fulling Fulling is an AI-powered full-stack engineer agent platform that provides dedicated AI workspaces combining skills, files, memory, and runt… | 63 | 2445 | active |
| hitsz-ids/synthetic-data-generator SDG (Synthetic Data Generator) is a Python framework for generating high-quality synthetic structured tabular data using deep learning, GAN… | 75 | 2440 | active |
| erew123/alltalk_tts AllTalk TTS is a text-to-speech application built on the Coqui TTS engine, usable standalone or as an extension for Text-generation-webui, … | 41 | 2439 | active |
| CopilotKit/OpenGenerativeUI An open-source framework and showcase for building generative UI, where an AI agent (CopilotKit + LangChain Deep Agents) generates interact… | 59 | 2438 | active |
| lmnr-ai/index Index is an open-source browser agent that autonomously performs complex tasks on the web by turning any website into an accessible API. It… | 10 | 2438 | active |