Open-source llm-inference software
3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
3358 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| HammerGPT/Hyper-Alpha-Arena A self-hosted AI-powered crypto trading platform for Hyperliquid and Binance Futures perpetual contracts. It combines LLM-driven trading ag… | 51 | 1185 | active |
| Gentleman-Programming/gentleman-guardian-angel Gentleman Guardian Angel (gga) is a pure-Bash CLI tool that runs AI-powered code reviews on staged files via a git pre-commit hook, using a… | 71 | 1184 | active |
| zjunlp/LightMem LightMem is a lightweight Python framework for adding long-term memory to LLM applications and AI agents, providing memory storage, retriev… | 61 | 1183 | active |
| AGenUI/AGenUI AGenUI is a high-performance A2UI SDK written in C++ that renders AI-generated, streaming user interfaces as native components on iOS, Andr… | 81 | 1182 | active |
| orpatashnik/StyleCLIP Official implementation of StyleCLIP, a method for text-driven manipulation of StyleGAN-generated imagery using CLIP. It provides three app… | 32 | 4117 | maintenance |
| nari-labs/dia2 Dia2 is a streaming dialogue text-to-speech model from Nari Labs that can generate conversational audio in realtime without needing the ful… | 38 | 1180 | active |
| shawwn/llama-dl A shell script that provides high-speed curl-based downloads of Meta's LLaMA model weights (7B, 13B, 30B, 65B), fetching all weights in und… | 31 | 4110 | maintenance |
| Facico/Chinese-Vicuna Chinese-Vicuna is a low-resource LLaMA+LoRA solution for building Chinese instruction-following language models, structured after Alpaca. I… | 34 | 4106 | maintenance |
| baichuan-inc/Baichuan2 Baichuan 2 is a family of open large language models (7B and 13B, Base and Chat variants with 4-bit quantized versions) trained by Baichuan… | 28 | 4082 | maintenance |
| i-am-bee/agentstack Agent Stack is open-source infrastructure (hosted by the Linux Foundation) for turning AI agents into running HTTP services, built on the A… | 78 | 1172 | active |
| whyiyhw/chatgpt-wechat A self-hosted Go application that lets users safely use LLM assistants (ChatGPT, Gemini, DeepSeek, Dify workflows) inside WeChat by relayin… | 56 | 1172 | active |
| nv-tlabs/LLaMA-Mesh LLaMA-Mesh is a fine-tuned large language model from NVIDIA Research that generates and understands 3D meshes by representing vertex coordi… | 24 | 1172 | active |
| kleneway/pastemax PasteMax is a cross-platform desktop application for browsing a repository's files, selecting relevant ones, and copying their contents for… | 56 | 1171 | active |
| Tencent/hpc-ops HPC-Ops is a production-grade C++/CUDA operator library for high-performance LLM inference, developed by Tencent's Hunyuan AI Infra team. I… | 58 | 1170 | active |
| BiFangKNT/mtga MTGA is a local proxy application that lets IDEs with fixed model providers (such as Trae) use arbitrary LLM backends. It exposes an OpenAI… | 70 | 1169 | active |
| espressif/esp-dl ESP-DL is Espressif's lightweight neural network inference framework for ESP-series chips, with a custom .espdl model format, quantization … | 69 | 1168 | active |
| IST-DASLab/marlin Marlin is a highly optimized FP16xINT4 matrix multiplication CUDA kernel for LLM inference that achieves near-ideal 4x speedups at batch si… | 27 | 1164 | active |
| chrishayuk/larql LARQL decompiles transformer models into a queryable 'vindex' (vector index) format and provides LQL, a SQL-like query language, to browse,… | 79 | 1162 | active |
| ricklamers/shell-ai Shell-AI (shai) is a Python CLI utility that converts natural language descriptions into shell command suggestions using LLMs via LangChain… | 44 | 1161 | active |
| Libr-AI/OpenFactVerification Loki is an open-source Python tool that automates fact verification by decomposing texts into claims, retrieving evidence via search, and u… | 16 | 1159 | active |
| character-ai/prompt-poet Prompt Poet is a Python library for designing LLM prompts using YAML and Jinja2 templates in a low-code approach. It renders dynamic, struc… | 50 | 1158 | active |
| Quorinex/Kiro-Go A Go service that converts Kiro accounts into OpenAI- and Anthropic-compatible API endpoints, with multi-account pooling, round-robin load … | 55 | 1157 | active |
| nico-martin/gemma4-browser-extension A Chrome extension that runs an on-device AI agent powered by Transformers.js and Gemma 4 via WebGPU, entirely locally with no data sent to… | 57 | 1155 | active |
| AI-QL/tuui TUUI is an open-source desktop MCP client and local AI playground built with Vue3 and Electron, serving as a unified interface for Model Co… | 68 | 1154 | active |
| shyamsn97/mario-gpt MarioGPT is a Python library and finetuned GPT2 model that generates playable Super Mario Bros levels from text prompts. It accompanies the… | 22 | 1152 | active |
| apple/ml-clara CLaRa is Apple's open-source end-to-end Retrieval-Augmented Generation model that compresses documents into continuous latent representatio… | 59 | 1151 | active |
| liujuntao123/smart-mermaid Smart Mermaid is an AI-powered web application that converts natural language text into Mermaid diagram code and renders it as visual chart… | 38 | 1151 | active |
| KroMiose/nekro-agent NekroAgent is a cross-platform AI agent framework for multi-user chat scenarios that generates and executes code in sandboxed containers, w… | 83 | 1149 | active |
| Gen-Verse/LatentMAS LatentMAS is a training-free multi-agent reasoning framework that lets LLM agents collaborate in continuous latent space by passing hidden-… | 53 | 1149 | active |
| agent-topia/evolving_personality JPAF is a Python framework that gives LLM agents structured, evolving MBTI personalities based on Carl Jung's psychological types. It uses … | 45 | 1149 | active |
| LiuMengxuan04/MiniCode MiniCode is a lightweight terminal-based AI coding assistant that implements a Claude Code-like agent loop with tool calling, diff review, … | 59 | 1147 | active |
| mtymek/opencode-obsidian An Obsidian plugin that embeds the OpenCode AI assistant's web view directly into Obsidian's sidebar, launching it via Node.js child proces… | 57 | 1147 | active |
| tontinton/maki Maki is a terminal-based AI coding agent written in Rust, optimized for minimal context token usage with tools like tree-sitter file indexi… | 77 | 1146 | active |
| terobox/ChatGPT-API-Faucet The frontend code for ChatGPT API Faucet, a public-welfare website where AI developers can claim $1.00 of free API tokens every 24 hours fo… | 32 | 1146 | active |
| Codium-ai/AlphaCodium Official implementation of the AlphaCodium paper, a test-based, multi-stage, iterative flow for LLM code generation on competitive programm… | 27 | 3966 | maintenance |
| atiilla/GeoIntel GeoIntel is a Python tool that uses Google's Gemini API to estimate where a photo was taken through AI-powered geolocation analysis. It off… | 55 | 1144 | active |
| microsoft/onnxruntime-genai ONNX Runtime GenAI is a C++ library with Python, C#, C/C++, and Java APIs for running generative AI models (LLMs, Whisper, vision-language … | 94 | 1142 | active |
| massgen/MassGen MassGen is an open-source multi-agent scaling system that runs in the terminal, orchestrating multiple frontier LLM agents to work on the s… | 66 | 1142 | active |
| RUC-NLPIR/DeepAgent DeepAgent is a general reasoning agent framework that dynamically searches for and uses tools from large toolsets (e.g., 16,000+ RapidAPIs)… | 49 | 1142 | active |
| xiaochong/hi-kid HiKid is a free, open-source Electron desktop app that lets children in non-English-speaking countries practice English speaking and listen… | 48 | 1142 | active |
| KoboldAI/KoboldAI-Client KoboldAI Client is a browser-based front-end for AI-assisted writing and text adventure gameplay that connects to multiple local and remote… | 67 | 3953 | maintenance |
| jianjieyiban/JJYB_AI_VideoAutoCut JJYB_AI 智剪 is a local-first desktop AI video creation workbench that combines material analysis, smart shot segmentation, commentary script… | 80 | 1141 | active |
| DeepMyst/Mysti Mysti is a VS Code extension that orchestrates a team of AI coding agents from 12 providers (Claude Code, OpenAI Codex, Gemini, Copilot, Cl… | 72 | 1141 | active |
| OpenLAIR/dr-claw Dr. Claw is an open-source, model-agnostic AI research assistant that provides a desktop and mobile UI for Claude Code, Cursor CLI, Codex, … | 78 | 1139 | active |
| mrphrazer/reverser_ai ReverserAI is a Binary Ninja plugin that provides automated reverse engineering assistance using locally-hosted large language models runni… | 63 | 1139 | active |
| flagos-ai/FlagGems FlagGems is a high-performance operator library for large language models written in the Triton language, providing backend-neutral GPU ker… | 88 | 1137 | active |
| rockbenben/subtitle-translator A free, browser-based batch subtitle translation tool for .srt, .ass, .vtt, and .lrc files that strips timing locally and sends only dialog… | 85 | 1136 | active |
| EuniAI/Prometheus Prometheus is a knowledge-graph-driven AI agent platform that maps, understands, and repairs complex codebases through reasoning rather tha… | 63 | 1136 | active |
| HarryR/z80ai Z80-μLM is a 2-bit quantized micro language model designed to run on 8-bit Z80 processors with only 64KB of RAM, exportable as ~40KB CP/M .… | 46 | 1136 | active |
| HacxGPT-Official/HacxGPT-CLI HacxGPT-CLI is a Python command-line chat client for accessing uncensored, unrestricted language models, including HacxGPT's own paid API m… | 74 | 1135 | active |
| rayenfeng/riko_project Project Riko is an anime-themed conversational voice assistant that combines OpenAI's GPT for dialogue, GPT-SoVITS for voice synthesis, and… | 28 | 1134 | active |
| NativeMindBrowser/NativeMindExtension NativeMind is an open-source browser extension that provides a fully private, on-device AI assistant by connecting to local LLMs via Ollama… | 60 | 1133 | active |
| openresponses/openresponses Open Responses is an open-source specification for multi-provider, interoperable LLM interfaces modeled on the OpenAI Responses API, defini… | 61 | 1132 | active |
| Dobiasd/frugally-deep frugally-deep is a lightweight header-only C++ library for running inference (forward passes) on Keras/TensorFlow models without linking ag… | 79 | 1128 | active |
| mrwadams/stride-gpt STRIDE GPT is an AI-powered threat modeling tool that uses LLMs to generate STRIDE-based threat models, attack trees, and MITRE ATT&CK/ATLA… | 80 | 1127 | active |
| futantan/OpenGpt OpenGpt is an open-source web platform for instantly running and creating small ChatGPT-powered prompt applications, with a browsable marke… | 39 | 3888 | maintenance |
| nnsight nnsight is a Python library for interpreting and intervening on the internals of deep learning models, built on PyTorch. It lets researcher… | 96 | 1125 | active |
| m1guelpf/chatgpt-telegram A Go application that runs a Telegram bot letting users chat with OpenAI's ChatGPT from a single command or Docker container. It handles au… | 23 | 3881 | maintenance |
| nobodywho-ooo/nobodywho NobodyWho is an open-source (EUPL-1.2) on-device LLM inference engine written in Rust, built on llama.cpp, with SDKs for Kotlin, Swift, Pyt… | 83 | 1122 | active |
| PromtEngineer/Verbi Verbi is a modular Python voice assistant application for experimenting with state-of-the-art transcription, LLM response generation, and t… | 45 | 1122 | active |
| plasma-umass/ChatDBG ChatDBG is an AI-assisted debugging tool that integrates large language models into standard debuggers (pdb, lldb, gdb) for C/C++/Python/Ru… | 65 | 1121 | active |
| Deep-ai-inc/ch.at A lightweight LLM chat service in Go, deployable as a single static binary, accessible via web, curl, SSH, DNS TXT queries, and an OpenAI-c… | 62 | 1121 | active |
| vamplabAI/sgr-agent-core SGR Agent Core is an open-source Python framework for building research agents using Schema-Guided Reasoning, with an extensible BaseAgent … | 79 | 1120 | active |
| XYZ-AI-Lab/AxisAgentic AxisAgentic is an extensible Python runtime for long-horizon AI agents that handles multi-turn execution, tool orchestration, context manag… | 52 | 1120 | active |
| KoljaB/RealtimeVoiceChat A Python client-server application enabling natural spoken conversations with LLMs, streaming browser audio via WebSockets through Realtime… | 30 | 3857 | maintenance |
| prometheus-eval/prometheus-eval Prometheus-Eval is a Python library for evaluating LLM generation outputs using the Prometheus family of open evaluator models and GPT-4 as… | 20 | 1117 | active |
| Doriandarko/make-it-heavy A Python framework that emulates Grok Heavy-style deep analysis by orchestrating multiple specialized AI agents in parallel via OpenRouter'… | 29 | 1116 | active |
| Code Interpreter API A Python library providing an open-source implementation of the ChatGPT Code Interpreter, built on LangChain. It lets LLM agents execute sa… | 20 | 3844 | maintenance |
| Kieirra/murmure Murmure is a privacy-first, open-source desktop speech-to-text application that transcribes voice entirely on-device using NVIDIA's Parakee… | 83 | 1114 | active |
| boona13/image-extender An open-source Next.js web app for AI image outpainting that extends images in any direction using Gemini models via OpenRouter, with Poiss… | 49 | 1113 | active |
| yeahhe365/Gemini-Nexus Gemini Nexus is a Chrome extension (Manifest V3) that adds an AI assistant layer to the browser, integrating Gemini Web, the Gemini API, an… | 79 | 1112 | active |
| index-labs/readpilot Read Pilot is a web application that analyzes online articles and generates Q&A cards from them using OpenAI models. It is built with Next.… | 60 | 1111 | active |
| dp-archive/archive Skill Compose is an open-source agent builder and runtime platform for creating skill-powered AI agents, written in Python. It deliberately… | 54 | 1111 | active |
| brenpoly/be-more-agent An offline-first conversational AI agent that turns a Raspberry Pi into a fully local voice assistant. It combines OpenWakeWord, Whisper.cp… | 59 | 1110 | active |
| gabber-dev/gabber Gabber is an open-source engine for building real-time multimodal AI applications that can see, hear, and speak, using graph-based orchestr… | 41 | 1110 | active |
| yoheinakajima/babyagi BabyAGI is an experimental Python framework for building self-building autonomous agents, built around a 'functionz' system that stores, ma… | 56 | 22363 | experimental |
| jina-ai/discoart DiscoArt is a Python library that wraps Disco Diffusion (CLIP-guided diffusion) so generative artists and developers can create AI artworks… | 23 | 3818 | maintenance |
| TheAiSingularity/graphrag-local-ollama A local adaptation of Microsoft's GraphRAG that runs LLM and embedding extraction entirely through Ollama models like llama3, mistral, gemm… | 57 | 1108 | active |
| google-ai-edge/litert-torch LiteRT Torch is a Python library that converts PyTorch models into .tflite format for running with LiteRT on Android, iOS, and IoT devices.… | 86 | 1107 | active |
| orailnoor/cross-platform-llm-client PrivateLM is a cross-platform AI chat client built with Flutter that unifies local on-device LLM inference (GGUF models with Vulkan GPU acc… | 67 | 1105 | active |
| alishobeiri/thread-notebook Thread is an AI-powered Jupyter Notebook alternative that integrates an AI copilot into the notebook editing experience. It runs locally an… | 56 | 1105 | active |
| PharMolix/OpenBioMed OpenBioMed is an open-source toolkit and agent platform for biomedicine and life science, offering multimodal models like BioMedGPT-Mol, 45… | 69 | 1104 | active |
| wilpel/caveman-compression A Python tool that compresses text for LLM contexts by stripping predictable grammar while preserving factual content, reducing token usage… | 38 | 1103 | active |
| jackdark425/aigroupapp AI Group is an open-source Android mobile assistant app that integrates multiple large language models such as OpenAI, Claude, Gemini, ERNI… | 36 | 1103 | active |
| vocodedev/vocode-core Vocode is an open-source Python library for building real-time, voice-based LLM applications and agents. It orchestrates streaming transcri… | 22 | 3797 | maintenance |
| vifreefly/kimuraframework Kimuraframework (Kimurai) is a Ruby web scraping framework with an AI-assisted DSL: an LLM generates XPath selectors from a schema on first… | 57 | 1102 | active |
| paean-ai/deeptide DeepTide is an AI coding agent tuned for DeepSeek models, shipped as a Swift-native macOS app plus cross-platform terminal CLIs (TypeScript… | 55 | 1102 | active |
| synth-inc/onit Onit SideKick is an open-source macOS AI chat sidebar that docks to any application, letting users chat with LLMs using highlighted text or… | 76 | 1100 | active |
| MrSibe/KnowNote KnowNote is a local-first, open-source desktop application built with Electron that serves as an alternative to Google NotebookLM. It lets … | 52 | 1100 | active |
| cbh123/stickerbaker StickerBaker is an open-source web application that generates AI stickers from text prompts or face uploads, powered by Replicate models (f… | 26 | 1100 | active |
| YOOTeam/OpenPPT OpenPPT is a web-based online presentation (PPT) editor based on ChatPPT, supporting the full workflow of creating, importing, editing, bea… | 33 | 1099 | active |
| xiaobright/dsh-anchored-standard A community plugin providing DeepSeek Harness (DSH) agent presets that bootstrap a session with a minimal tool schema and then promote to a… | 55 | 3777 | maintenance |
| welltop-cn/ComfyUI-TeaCache A ComfyUI plugin integrating TeaCache, a training-free caching method that accelerates diffusion model inference by exploiting output diffe… | 32 | 1097 | active |
| xataio/agent Xata Agent is an open-source AI agent that acts as a PostgreSQL expert, monitoring database logs and metrics, diagnosing root causes of iss… | 10 | 1094 | active |
| hinthornw/trustcall A Python library that improves LLM tool calling reliability by having models generate JSON patch operations instead of full JSON blobs. It … | 35 | 1093 | active |
| yandex/YaLM-100B YaLM-100B is a GPT-like pretrained language model with 100 billion parameters, trained by Yandex on English and Russian text using DeepSpee… | 32 | 3756 | maintenance |
| spring-ai-alibaba/Lynxe Lynxe (formerly JManus) is a Java-based, code-free 'Prompt Programming' studio implementing a Manus-style multi-agent system with high exec… | 70 | 1090 | active |
| NVlabs/Fast-dLLM NVIDIA's official implementation of Fast-dLLM, a family of training-free and fine-tuning-based acceleration techniques for diffusion-based … | 54 | 1090 | active |
| HswAI2026/JuZhou-V1 JuZhou 1.0 is an ultra-lightweight 0.387B-parameter text-to-image foundation model designed for fully offline, on-device execution on mobil… | 51 | 1089 | active |
| piotrostr/listen Listen is a Rust toolkit and framework for algorithmic trading on Solana, evolving into a framework for AI-driven crosschain portfolio mana… | 44 | 1089 | active |