Open-source llm-inference software
3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
3358 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| nndeploy/nndeploy nndeploy is an easy-to-use, high-performance AI deployment framework written in C++ with Python bindings. It provides a visual drag-and-dro… | 81 | 1880 | active |
| nolanx-ai/nolanx.ai NolanX is an open-source multi-modal agent platform for AI filmmaking that orchestrates text, image, audio, and video models into long-runn… | 49 | 1877 | active |
| Jido Jido is an autonomous agent framework for Elixir that runs agents as supervised BEAM processes with fault tolerance, signal-based coordinat… | 81 | 1875 | active |
| THUDM/LongWriter LongWriter is a research project from Tsinghua's THUDM group providing fine-tuned LLMs (based on GLM-4-9B and Llama-3.1-8B) plus training a… | 33 | 1874 | active |
| dauparas/ProteinMPNN ProteinMPNN is a PyTorch-based tool that designs amino acid sequences for given protein backbone structures using a message-passing neural … | 23 | 1872 | stable |
| dphnAI/sonar Sonar is a large-scale LLM inference engine for Hugging Face-compatible language and multimodal models, based on vLLM (formerly the Aphrodi… | 92 | 1871 | active |
| TypeWhisper/typewhisper-mac TypeWhisper is a macOS application for local, on-device speech-to-text dictation built on WhisperKit, with optional cloud transcription via… | 79 | 1869 | active |
| jiujiu532/grok2api A self-hosted Python/FastAPI gateway that exposes grok.com and console.x.ai chat, image, and video capabilities through OpenAI- and Anthrop… | 52 | 1867 | active |
| MixLabPro/comfyui-mixlab-nodes A ComfyUI custom-nodes plugin that adds workflow-to-web-app conversion, screen sharing, floating video, GPT/LLM integration, 3D, speech rec… | 53 | 1862 | active |
| webpilot-ai/Webpilot Webpilot is a free, open-source browser extension that acts as a 'Copilot for the web', letting users converse with web page content withou… | 72 | 1859 | active |
| RanFeng/clipsketch-ai ClipSketch AI is a web-based AI content creation workbench that imports videos from Bilibili and Xiaohongshu links, lets users frame-accura… | 38 | 1858 | active |
| sunmh207/AI-Codereview-Gitlab A self-hosted automated code review application that uses large language models (DeepSeek, OpenAI, Anthropic, ZhipuAI, Qwen, Ollama) to rev… | 81 | 1857 | active |
| Tele-AI/Telechat TeleChat is a family of open-source bilingual (Chinese-English) large language models (1B, 7B, 12B) developed by China Telecom's AI team, r… | 64 | 1852 | active |
| Kav-K/GPTDiscord GPTDiscord is a self-hosted Discord bot providing an all-in-one GPT interface with ChatGPT-style conversations, DALL-E image generation, AI… | 100 | 1850 | active |
| HKUDS/OpenOPC OpenOPC is a Python framework for building a personal AI-native company that recruits role-specific AI employees, assigns and hands off tas… | 56 | 1848 | active |
| aallam/openai-kotlin A Kotlin client library for the OpenAI API built on Kotlin multiplatform and coroutines, using Ktor as the HTTP engine. It provides typed a… | 57 | 1847 | active |
| YangLing0818/RPG-DiffusionMaster Official implementation of RPG (Recaption, Plan, Generate), a training-free framework that uses multimodal LLMs as prompt recaptioners and … | 26 | 1842 | active |
| magenta/magenta-realtime Magenta RealTime 2 (MRT2) is an open-weights model for real-time, interactive music generation controllable via text, MIDI, and audio. It s… | 70 | 1839 | active |
| abinthomasonline/repo2txt A browser-based tool that converts GitHub, GitLab, Azure DevOps repositories, local folders, or ZIP files into a single formatted text file… | 63 | 1837 | active |
| trpc-group/trpc-agent-go tRPC-Agent-Go is a Go-native framework for building production LLM agent systems, offering agents, graph workflows (LangGraph-equivalent), … | 79 | 1836 | active |
| hyperfield/ai-file-sorter AI File Sorter is a cross-platform desktop application that uses local or remote LLMs to categorize files and suggest cleaner filenames for… | 85 | 1834 | active |
| he-yufeng/CoreCoder CoreCoder is a minimal AI coding agent written in roughly 1,000 lines of Python, inspired by Claude Code and designed to work with any LLM.… | 82 | 1833 | active |
| tile-ai/TileRT TileRT is a tile-based runtime for ultra-low-latency LLM inference, achieving hundreds to over 1000 tokens/s decode speeds on frontier mode… | 75 | 1831 | active |
| evalplus/evalplus EvalPlus is a Python framework and benchmark suite for rigorously evaluating LLM-synthesized code, providing HumanEval+ and MBPP+ datasets … | 37 | 1826 | active |
| Mellea Mellea is a Python library for writing generative programs, where LLM calls are first-class operations with type-annotated outputs, verifia… | 84 | 1821 | active |
| APIParkLab/APIPark APIPark is an open-source, cloud-native AI gateway and API developer portal for managing, integrating, and deploying AI and REST APIs. It u… | 86 | 1819 | active |
| potamides/DeTikZify DeTikZify is a Python library and research tool that uses multimodal large language models to synthesize TikZ/LaTeX graphics programs from … | 63 | 1819 | active |
| jacoblee93/fully-local-pdf-chatbot A Next.js web application that lets users chat with uploaded PDF documents entirely locally, performing chunking, embedding, vector storage… | 49 | 1817 | active |
| JHubi1/ollama-app A modern, easy-to-use Flutter client app for Ollama that connects to an Ollama server over the local network to chat with local LLMs. It ru… | 59 | 1816 | active |
| inference-labs-inc/inference-network Inference Network is a Proof of Inference system built on EigenLayer that provides trustless verification of AI inference results via Solid… | 47 | 1814 | active |
| sybil-solutions/local-studio Local Studio is a local-first desktop application for launching, managing, and chatting with self-hosted LLM backends like vLLM, SGLang, ll… | 76 | 1813 | active |
| delibae/claude-prism ClaudePrism is an open-source, offline-first desktop application for scientific and academic writing, built with Tauri 2 and Rust. It combi… | 72 | 1813 | active |
| SunoAI-API/Suno-API An unofficial Python/FastAPI-based API wrapper for Suno AI that lets you generate songs and lyrics programmatically. It includes automatic … | 30 | 1811 | active |
| AI Insight Daily A Cloudflare Workers-based content aggregation and generation platform that daily curates AI industry news, trending open-source projects, … | 64 | 1809 | active |
| undertheseanlp/underthesea Underthesea is an open-source Python toolkit for Vietnamese natural language processing that has evolved into an agentic AI toolkit with mu… | 91 | 1806 | active |
| High-Logic/Genie-TTS GENIE is a lightweight Python inference engine for the open-source GPT-SoVITS text-to-speech project, optimized for fast CPU-based speech s… | 58 | 1806 | active |
| Vitis AI AMD Vitis AI is an integrated development environment and stack for accelerating AI inference on AMD/Xilinx adaptable platforms, including … | 64 | 1805 | active |
| wm94i/Work-Review Work Review is a local-first desktop application that automatically tracks which apps you used, websites you visited, window titles, usage … | 79 | 1804 | active |
| granawkins/rawdog Rawdog is a CLI assistant that responds to natural-language prompts by generating and auto-executing Python scripts. It uses a recursive se… | 30 | 1803 | active |
| LM Studio SDK lmstudio-js is the official TypeScript/JavaScript client SDK for LM Studio, enabling use of locally running LLMs for chat, completions, emb… | 73 | 1799 | active |
| zai-org/CogVLM CogVLM is an open-source visual language model (17B) combining a vision encoder with a pretrained language model for image understanding an… | 28 | 6745 | maintenance |
| freestylefly/director_ai An AI-powered mobile application for creating comic-style short dramas, generating scripts, storyboards, and composed videos from a single … | 49 | 1797 | active |
| AI-Citizen/SolidGPT SolidGPT is an AI-powered semantic search assistant for developers that lets you chat with your codebase and Notion workspace. It ships as … | 19 | 1797 | active |
| OS-Copilot/OS-Copilot OS-Copilot is an open-source Python library for building generalist AI agents that interface with operating system elements like the web, t… | 17 | 1794 | active |
| QwenLM/Qwen-VL Official repository for Qwen-VL, Alibaba Cloud's large vision-language model family, including the pretrained Qwen-VL and instruction-tuned… | 29 | 6725 | maintenance |
| vercel-labs/coding-agent-template A deployable Next.js web application template for building multi-agent AI coding platforms that run coding agents like Claude Code, Codex C… | 59 | 1792 | active |
| hitsz-ids/airda airda (Air Data Agent) is a Python-based multi-agent system for data analysis that understands natural language data requirements and gener… | 17 | 1792 | active |
| iBUHub/AIStudioToAPI A self-hosted proxy service that wraps the Google AI Studio Build web interface into OpenAI, Gemini, and Anthropic compatible APIs using br… | 83 | 1789 | active |
| AtomicBot-ai/Atomic-Chat Atomic Chat is an open-source desktop and mobile application with a built-in inference engine for running open-weight LLMs fully locally an… | 82 | 1788 | active |
| 0xSero/turboquant TurboQuant is a Python library implementing near-optimal KV cache quantization for LLM inference, compressing keys to 3-bit and values to 2… | 57 | 1788 | active |
| octo-models/octo Octo is an open-source generalist robot policy: a transformer-based diffusion policy pretrained on 800k robot trajectories from the Open X-… | 18 | 1787 | active |
| anthropics/claude-agent-sdk-typescript The Claude Agent SDK is a TypeScript/Node.js library for programmatically building AI agents powered by Claude Code's capabilities. It enab… | 80 | 1785 | active |
| google-deepmind/concordia Concordia is a Python library for building generative agent-based social simulations, using a tabletop role-playing game pattern where a Ga… | 83 | 1782 | active |
| neuml/paperai paperai is an AI application for medical and scientific papers that runs bulk LLM inference and RAG pipelines over article repositories to … | 63 | 1781 | active |
| aieditor-team/AiEditor AiEditor is an open-source, AI-powered rich text WYSIWYG editor built on Web Components (ProseMirror/Tiptap-based) that integrates with any… | 72 | 1778 | active |
| FSoft-AI4Code/CodeWiki CodeWiki is an open-source Python framework that uses LLM-powered multi-agent processing to generate holistic, structured documentation for… | 58 | 1778 | active |
| ExtensityAI/symbolicai SymbolicAI is a Python neuro-symbolic framework that blends classical programming with LLM capabilities through composable Symbol primitive… | 93 | 1777 | active |
| Emu Series Emu3 is a suite of state-of-the-art multimodal models from BAAI trained solely with next-token prediction, tokenizing images, text, and vid… | 53 | 1777 | active |
| wwbin2017/bailing Bailing is an open-source voice assistant application similar to GPT-4o, built with an ASR + VAD + LLM + TTS pipeline (FunASR, silero-vad, … | 51 | 1775 | active |
| Nutlope/twitterbio A Next.js web application that generates Twitter/X bios using AI models (Qwen 3.5 9B and GPT OSS 20B) served via the Together AI API with s… | 68 | 1770 | active |
| crazyykhllc-bit/CyberPPT CyberPPT is a Codex Skill that converts documents, research materials, and business data into high-density, editable, consulting-style Powe… | 52 | 1770 | active |
| zubair-trabzada/ai-legal-claude A Claude Code skill pack providing an AI legal assistant with 14 slash commands and 5 parallel agents for contract review, risk analysis, N… | 45 | 1768 | active |
| jenslys/opencode-gemini-auth A TypeScript plugin for the Opencode CLI that authenticates with Google via OAuth so users can use their existing Gemini plan and quotas (i… | 82 | 1767 | active |
| adamlyttleapps/claude-skill-aso-appstore-screenshots A Claude Code skill that generates high-converting App Store screenshots for iOS apps by analyzing the codebase to identify core benefits a… | 45 | 1767 | active |
| AgentDock/AgentDock AgentDock is an open-source, backend-first TypeScript framework for building AI agents with configurable determinism, plus a Next.js refere… | 68 | 1766 | active |
| steven-jianhao-li/zotero-AI-Butler A Zotero plugin that acts as an AI-powered research butler: it automatically scans PDFs added to your Zotero library and sends them to a us… | 77 | 1765 | active |
| AIdea AIdea is a fully open-source cross-platform mobile and desktop app built with Flutter that integrates mainstream large language models (GPT… | 64 | 1765 | active |
| signerlabs/Klee Klee is a native macOS AI chat application that runs large language models entirely on-device using Apple's MLX framework on Apple Silicon.… | 50 | 1765 | active |
| GoogleCloudPlatform/agent-starter-pack A Python package from Google Cloud Platform that provides production-ready templates for building and deploying GenAI agents on Google Clou… | 68 | 6569 | maintenance |
| webonnx/wonnx Wonnx is a GPU-accelerated ONNX inference runtime written entirely in Rust, built on wgpu and usable natively or in the browser via WebGPU … | 10 | 1754 | active |
| cubewhy/skid-homework A browser-based, AI-powered homework solver built with Next.js that sends images and PDFs of homework problems to a Gemini or OpenAI-compat… | 60 | 1752 | active |
| facebookresearch/metaseq Metaseq is a PyTorch codebase from Meta AI for training and working with large-scale Open Pre-trained Transformers (OPT), forked from fairs… | 10 | 6544 | maintenance |
| software-mansion/react-native-executorch React Native ExecuTorch is a declarative React Native library for running AI models on-device, powered by Meta's ExecuTorch runtime. It shi… | 88 | 1750 | active |
| HKUDS/Auto-Deep-Research Auto-Deep-Research is an open-source, cost-efficient alternative to OpenAI's Deep Research, built on the AutoAgent framework. It runs as a … | 38 | 1750 | active |
| Stability-AI/StableCascade Official codebase for Stable Cascade, a text-to-image generation model built on the Würstchen architecture with a highly compressed latent … | 26 | 6528 | maintenance |
| bespokelabsai/curator Bespoke Curator is a Python library for bulk LLM inference and scalable synthetic data curation for post-training and structured data extra… | 89 | 1744 | active |
| yincongcyincong/MuseBot MuseBot is a self-hosted Go chatbot application that connects messaging platforms (Telegram, Discord, Slack, Lark/Feishu, DingTalk, WeCom, … | 84 | 1743 | active |
| 14790897/paper-ai A web application that helps users write academic papers with AI while searching real literature from Semantic Scholar, arXiv, and PubMed a… | 64 | 1740 | active |
| Xiaojiu-z/EasyControl EasyControl is the official implementation of an ICCV 2025 paper adding efficient and flexible conditional control to Diffusion Transformer… | 32 | 1738 | active |
| kwaroran/Risuai RisuAI is a cross-platform AI roleplay and character chatting application built with Svelte, TypeScript, and Tauri. It acts as a frontend f… | 91 | 1737 | active |
| Gitlawb/zero Zero is a terminal-based AI coding agent written in Go that can inspect repositories, edit files, run commands, and manage durable local se… | 80 | 1736 | active |
| jesselau76/ebook-GPT-translator A Python CLI and GUI tool that translates ebooks (TXT, EPUB, DOCX, PDF, optional MOBI) using LLM providers like OpenAI, Azure, Codex, Claud… | 60 | 1735 | active |
| muscriptor/muscriptor MuScriptor is a multi-instrument music transcription model by Kyutai and Mirelo that converts audio recordings into MIDI and sheet music. I… | 78 | 1732 | active |
| Codex CLI Integration A ZSH plugin that brings AI-powered code completion to the command line, supporting OpenAI Codex, Google Gemini, and other LLM providers. U… | 34 | 1732 | active |
| NVIDIA/FasterTransformer NVIDIA's highly optimized C++/CUDA library for fast inference of Transformer-based models such as BERT, GPT, and encoder-decoder models, wi… | 23 | 6450 | maintenance |
| AvaLovelace1/BrickGPT BrickGPT is the official implementation of an ICCV 2025 Best Paper approach that generates physically stable, buildable toy brick (LEGO) mo… | 54 | 1726 | active |
| LLPhant/LLPhant LLPhant is a comprehensive PHP generative AI framework inspired by LangChain and LlamaIndex, providing tools for working with LLMs, embeddi… | 96 | 1722 | active |
| fish2018/YPrompt YPrompt is a self-hosted web application that uses AI-guided conversation to elicit user requirements and automatically generate profession… | 41 | 1721 | active |
| lotus-data/lotus LOTUS is a Python library providing a Pandas-like API of LLM-powered semantic operators (map, filter, extract, aggregate, top-k) for bulk p… | 85 | 1719 | active |
| tenstorrent/tt-metal TT-Metal is Tenstorrent's open-source software stack containing TT-NN, a Python and C++ neural network operator library, and TT-Metalium, a… | 99 | 1718 | active |
| mithun50/openclaw-termux OpenClaw is an AI gateway that can be run on Android as a standalone Flutter app with a built-in terminal, web dashboard, and one-tap setup… | 79 | 1717 | active |
| LLMUnity LLMUnity is an open-source Unity plugin that runs large language models locally in Unity games via llama.cpp, enabling AI-driven characters… | 70 | 1716 | active |
| glidea/zenfeed zenfeed is a self-hosted AI-powered RSS reader and information hub built in Go. It aggregates RSS feeds, uses LLMs to filter, summarize, an… | 62 | 1716 | active |
| kingoflolz/mesh-transformer-jax A JAX/Haiku library implementing model-parallel training and inference of transformer models using xmap/pjit operators, similar to Megatron… | 32 | 6384 | maintenance |
| MorDavid/BruteForceAI BruteForceAI is a Python-based penetration testing tool that uses LLMs (via Ollama or Groq) to automatically analyze login page HTML and id… | 56 | 1715 | active |
| langchain-ai/langmem LangMem is a Python library that helps AI agents learn and adapt from their interactions over time by extracting information from conversat… | 65 | 1713 | active |
| xhongc/ai_story AI Story is a self-hosted platform that automates story video production: given a topic, it generates scripts, storyboards, AI images, came… | 60 | 1713 | active |
| gi-dellav/zerostack zerostack is a minimal coding agent written in Rust, optimized for a tiny memory footprint (~16 MB RAM) and fast performance. It provides a… | 81 | 1708 | active |
| nat/openplayground A self-hosted LLM playground that runs on your laptop, providing a web UI for prompting models from OpenAI, Anthropic, Cohere, HuggingFace,… | 57 | 6348 | maintenance |
| xingyaoww/code-act CodeAct is a research framework and agent system that uses executable Python code as a unified action space for LLM agents, enabling multi-… | 27 | 1705 | active |