Open-source llm-inference software
3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
3358 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| LiteLLM LiteLLM is an open-source AI gateway and Python SDK that provides a unified OpenAI-format interface to 100+ LLM providers (OpenAI, Anthropi… | 92 | 61456 | active |
| MemPalace/mempalace MemPalace is a local‑first AI memory system that stores conversation and project data verbatim and makes it searchable via semantic search.… | 82 | 59690 | active |
| CLIProxyAPI A Go-based proxy server that exposes OpenAI/Gemini/Claude/Codex-compatible API endpoints backed by CLI AI assistant accounts (Claude Code, … | 81 | 54756 | active |
| BMad Method BMad Method is an open‑source framework that guides agile AI‑driven software development from idea to working code. It provides CLI tools, … | 87 | 54038 | active |
| Academic Research Skills Academic Research Skills is a Python library of Claude Code skills that guides researchers through the full academic workflow, from questio… | 83 | 52555 | active |
| LocalAI LocalAI is an open‑source AI runtime that provides a single binary exposing OpenAI‑compatible APIs for text, vision, audio, and video model… | 94 | 49857 | active |
| LocalAI LocalAI is an open-source, self-hosted AI inference runtime that runs LLMs, vision, speech, image, and video models behind OpenAI/Anthropic… | 94 | 49857 | active |
| oobabooga/textgen TextGen is an open-source desktop application for running large language models locally with full privacy and no telemetry. It provides a c… | 83 | 47700 | active |
| LibreChat LibreChat is an open-source, self-hosted web application providing a unified ChatGPT-style interface for multiple AI providers (OpenAI, Ant… | 85 | 46021 | active |
| vectorize-io/hindsight Hindsight is an open-source agent memory system that gives AI agents persistent long-term memory that learns and consolidates knowledge ove… | 78 | 44347 | active |
| alibaba/open-code-review OpenCodeReview is a hybrid architecture code review tool that combines deterministic pipelines with LLM agents to provide precise line‑leve… | 76 | 44064 | active |
| ray-project/ray Ray is a unified open-source framework for scaling AI and Python applications, consisting of a core distributed runtime (tasks, actors, obj… | 99 | 44057 | stable |
| Agno Agno is a Python‑based SDK and runtime for building, deploying, and managing AI agent platforms. It provides a full stack—from agent defini… | 95 | 42675 | active |
| lfnovo/open-notebook Open Notebook is an open-source, privacy-focused alternative to Google's Notebook LM that lets users organize multi-modal sources (PDFs, vi… | 90 | 40397 | active |
| HKUDS/LightRAG LightRAG is a Python library that implements a simple and fast Retrieval-Augmented Generation framework using graph structures and vector e… | 84 | 40153 | active |
| 2noise/ChatTTS ChatTTS is a generative text-to-speech model optimized for conversational dialogue scenarios such as LLM assistants, supporting English and… | 64 | 39885 | active |
| Project NOMAD Project NOMAD is a self-hosted, offline-first knowledge and education server that packages Wikipedia, books, courses, offline maps, and opt… | 85 | 39414 | active |
| chatchat-space/Langchain-Chatchat Langchain-Chatchat is an open-source, offline-deployable RAG and Agent application built on Langchain and LLMs such as ChatGLM, Qwen, and L… | 41 | 38672 | active |
| khoj-ai/khoj Khoj is a self-hostable personal AI assistant ('second brain') that answers questions from your own documents and the web using local or on… | 83 | 37738 | active |
| Langfuse Langfuse is an open-source AI engineering platform for LLM observability, tracing, evaluation, prompt management, and metrics, deployable a… | 94 | 36043 | active |
| changedetection.io A self-hosted web application for monitoring websites and detecting content changes, with alerts via Discord, Slack, Telegram, email, webho… | 94 | 34909 | active |
| OpenBMB/ChatDev ChatDev 2.0 (DevAll) is a zero‑code multi‑agent platform that lets users configure and run complex LLM‑driven workflows without writing cod… | 74 | 34379 | active |
| spaCy spaCy is an industrial-strength natural language processing library for Python and Cython, featuring pretrained pipelines for 70+ languages… | 97 | 33952 | stable |
| Fish Speech Fish Speech is an open-source state-of-the-art text-to-speech system (Fish Audio S2) trained on over 10 million hours of audio across ~50 l… | 70 | 33062 | active |
| zeroclaw-labs/zeroclaw ZeroClaw is a lightweight, single‑binary personal AI assistant written in Rust. It runs on any OS, connects to dozens of LLM providers and … | 81 | 32990 | active |
| AIHawk AIHawk is an open-source Python AI agent that automates job applications by scraping job postings via browser automation and auto-applying … | 87 | 32150 | active |
| Zep Graphiti is an open-source Python framework for building and querying temporal knowledge graphs (Context Graphs) for AI agents. It incremen… | 87 | 31991 | active |
| BigPizzaV3/CodexPlusPlus Codex++ is an open-source external launcher and management tool for the OpenAI Codex / ChatGPT desktop apps, built with Rust and Tauri. It … | 77 | 31876 | active |
| mksglu/context-mode An open-source MCP plugin that optimizes AI coding agent context windows by intercepting large tool outputs, storing them in a local FTS5 i… | 73 | 31846 | active |
| XiaoZhi AI Chatbot XiaoZhi is an open-source MCP-based AI voice chatbot firmware for ESP32-family microcontrollers, connecting large language models like Qwen… | 91 | 31445 | active |
| ScrapeGraphAI ScrapeGraphAI is a Python library that leverages large language models to automate web scraping and data extraction. It provides a high‑lev… | 87 | 31358 | active |
| Cognee Cognee is an open‑source AI memory platform that provides persistent, long‑term memory for AI agents. It builds a knowledge graph from docu… | 90 | 31294 | active |
| Cognee Cognee is an open-source Python library and platform that gives AI agents persistent long-term memory by ingesting data in any format and b… | 90 | 31294 | active |
| GBrain gstack is a TypeScript‑based CLI that turns Claude Code into a virtual engineering team of AI agents. It provides 23 opinionated tools that… | 78 | 30972 | active |
| GBrain GBrain is a self-hosted 'brain layer' for AI agents that combines a knowledge graph, vector search, and a synthesis engine to return cited … | 78 | 30972 | active |
| GPT Researcher GPT Researcher is an open-source autonomous deep research agent that plans research questions, scrapes web and local sources, and produces … | 95 | 29943 | active |
| karakeep-app/karakeep Karakeep is a self‑hostable "bookmark‑everything" application that lets you save links, notes, images and PDFs. It adds AI‑powered automati… | 94 | 29774 | active |
| Mastra Mastra is a TypeScript framework for building AI-powered applications and autonomous agents, with model routing across 40+ providers, graph… | 87 | 28986 | active |
| ifixai-ai/iFixAi iFixAi is an open-source CLI diagnostic tool that independently audits AI agents for misalignment, hallucination, manipulation, deception, … | 82 | 28773 | active |
| ATH-MaaS/Pixelle-Video Pixelle-Video is an AI-powered fully automated short video engine that turns a single topic into a finished video. It automatically writes … | 61 | 28735 | active |
| xai-org/grok-build Grok Build is SpaceXAI's terminal-based AI coding agent, written in Rust, that runs as a full-screen interactive TUI. It understands codeba… | 57 | 27197 | active |
| Haystack Haystack is an open-source Python AI orchestration framework by deepset for building production-ready LLM applications, agents, and RAG sys… | 98 | 26720 | stable |
| resemble-ai/chatterbox Chatterbox is a family of state-of-the-art open-source text-to-speech models by Resemble AI, including multilingual and low-latency Turbo v… | 49 | 26574 | active |
| zai-org/Open-AutoGLM Open-AutoGLM is an open phone agent model and framework built on AutoGLM that understands phone screens multimodally and automates Android … | 45 | 26342 | active |
| Cinnamon/kotaemon Kotaemon is an open-source RAG-based application providing a clean, customizable UI for chatting and doing question answering with your own… | 78 | 25795 | active |
| nextai-translator/nextai-translator A browser extension and cross-platform desktop application that provides translation of selected text using the ChatGPT/OpenAI API. It also… | 92 | 24977 | active |
| index-tts/index-tts IndexTTS is an industrial-level zero-shot text-to-speech system that clones a voice from a single reference audio clip. The latest IndexTTS… | 76 | 24518 | active |
| PaddlePaddle PaddlePaddle is an industrial-grade deep learning framework written in C++ with Python APIs, supporting high-performance single-machine and… | 91 | 24105 | active |
| QwenAudio/CosyVoice CosyVoice is a multilingual large voice generation model (TTS) with full-stack inference, training, and deployment support. It offers zero-… | 58 | 23763 | active |
| facebookresearch/audiocraft A PyTorch library from Meta for audio processing and generation with deep learning, featuring the EnCodec neural audio codec and generative… | 58 | 23699 | active |
| 1Panel-dev/MaxKB MaxKB is an open-source, self-hosted platform for building enterprise-grade AI agents with RAG pipelines, workflow orchestration, and MCP t… | 96 | 22992 | active |
| Opik Opik is an open-source LLM observability and evaluation platform by Comet for tracing, evaluating, and monitoring LLM applications, RAG pip… | 93 | 22726 | active |
| wavetermdev/waveterm Wave Terminal is an open-source, cross-platform terminal emulator with integrated AI assistance, durable SSH sessions, file previews, a bui… | 89 | 22405 | active |
| chidiwilliams/buzz Buzz is a cross-platform desktop application that transcribes and translates audio and video offline using OpenAI's Whisper models. It offe… | 91 | 21899 | active |
| jina-ai/serve Jina-serve is a Python framework for building and deploying multimodal AI services that communicate via gRPC, HTTP, and WebSockets. It prov… | 36 | 21854 | active |
| Eclipse Theia Eclipse Theia is an open-source framework written in TypeScript for building custom cloud (browser-based) and desktop IDEs and tools, with … | 98 | 21705 | active |
| coze-dev/coze-studio Coze Studio is an open-source, all-in-one AI agent development platform with visual, no-code/low-code tools for building, debugging, and de… | 67 | 21698 | active |
| vercel/chatbot Chatbot is an open‑source, full‑featured template built with Next.js and the Vercel AI SDK. It provides a ready‑to‑deploy AI chat applicati… | 67 | 20983 | active |
| LLM Wiki LLM Wiki is a cross-platform desktop application that automatically turns your documents into an organized, interlinked knowledge base. Unl… | 79 | 20905 | active |
| SETA OWL is a multi-agent collaboration framework built on the CAMEL-AI framework for automating real-world tasks through dynamic agent interact… | 65 | 20153 | active |
| dzhng/deep-research An AI-powered deep research agent that iteratively generates search queries, scrapes web content, and refines its research direction using … | 52 | 19779 | active |
| state-spaces/mamba Mamba is a PyTorch library implementing the selective state space model (SSM) architecture for linear-time sequence modeling, with hardware… | 80 | 18874 | active |
| microsoft/agent-lightning Agent Lightning is a lightweight (~3,500-line) agentic reinforcement learning framework from Microsoft for training LLM-powered agents with… | 86 | 18782 | active |
| NVIDIA-NeMo/Speech NVIDIA NeMo Speech is an open‑source Python toolkit for building, training, and deploying speech AI models such as automatic speech recogni… | 94 | 18521 | active |
| browser-use/browser-harness Browser Harness is a self-healing Python harness that connects LLM agents directly to a real browser via a single editable CDP websocket. A… | 78 | 18360 | active |
| arc53/DocsGPT DocsGPT is an open-source AI platform for building private agents, assistants, and enterprise search over your own documents. It includes a… | 97 | 18312 | active |
| LangBot LangBot is an open-source, production-grade platform for building AI-powered instant messaging bots, connecting LLMs and agent workflows to… | 95 | 18291 | active |
| RightNow-AI/openfang OpenFang is an open-source Agent Operating System written in Rust that runs autonomous AI agents on schedules, 24/7, with pre-built capabil… | 63 | 18208 | active |
| rowboatlabs/rowboat Rowboat is an open-source desktop AI coworker that indexes your email, meetings, Slack, and conversations into a living backlinked knowledg… | 86 | 17974 | active |
| rocketride-org/rocketride-server RocketRide is an open-source AI pipeline engine with a high-throughput C++ runtime and 100+ Python-extensible nodes for building, debugging… | 82 | 17760 | active |
| Wan Wan is an open-source family of large-scale video generation models (Wan2.1/Wan2.2) with Python inference code and pretrained weights for t… | 61 | 17695 | active |
| lllyasviel/FramePack FramePack is a next-frame-prediction video diffusion framework and desktop application that generates long videos progressively while compr… | 28 | 17304 | active |
| langchain-ai/openwiki OpenWiki is a TypeScript CLI that uses a Deep Agents documentation agent to generate and continuously maintain a linked Markdown wiki for a… | 81 | 17281 | active |
| MemoriLabs/Memori Memori is agent-native memory infrastructure that turns LLM agent execution and conversation into structured, persistent state. It ships as… | 77 | 17149 | active |
| QwenLM/Qwen-Agent Qwen-Agent is a Python framework for building LLM agent applications on top of Qwen models, providing function calling, MCP support, code i… | 58 | 17128 | active |
| Elysia Weaviate is an open-source, cloud-native vector database written in Go that stores both objects and vector embeddings, combining vector sim… | 93 | 16860 | stable |
| memvid/memvid Memvid is a single-file memory layer for AI agents that packages data, embeddings, and search indexes into one portable .mv2 file. It repla… | 72 | 16606 | active |
| Pipecat Pipecat is an open-source Python framework (BSD-2) for building real-time voice and multimodal conversational AI agents. It orchestrates sp… | 89 | 16579 | active |
| Trigger.dev Trigger.dev is an open-source TypeScript platform for building and deploying durable background jobs, long-running AI workflows, and AI age… | 94 | 16478 | active |
| KittenML/KittenTTS KittenTTS is an open-source, ultra-lightweight text-to-speech library built on ONNX, with models ranging from 15M to 80M parameters (25-80 … | 65 | 16161 | active |
| tracel-ai/burn Burn is a Rust-based tensor library and deep learning framework supporting training and inference through a unified API. It JIT-compiles te… | 86 | 16008 | active |
| chatfire-AI/huobao-drama Huobao Drama is a self-hosted, AI-powered end-to-end short drama generation platform that automates the full pipeline from a one-sentence i… | 82 | 15921 | active |
| vibrantlabsai/ragas Ragas is a Python library for evaluating LLM applications with objective, LLM-driven metrics and systematic experiment loops. It also gener… | 65 | 15901 | active |
| coderamp-labs/gitingest Gitingest turns any Git repository into a prompt-friendly text digest of its codebase, suitable for feeding into LLMs. It is available as a… | 68 | 15879 | active |
| GaiZhenbiao/ChuanhuChatGPT Chuanhu Chat is a Python/Gradio-based web GUI for chatting with ChatGPT and many other LLMs (GPT-5/4, Claude, Gemini, ChatGLM, Llama, Ollam… | 64 | 15269 | active |
| NanmiCoder/cc-haha A local-first cross-platform desktop workspace for Claude Code and AI coding agents, built with Electron, React, and TypeScript. It provide… | 83 | 15120 | active |
| midday-ai/midday Midday is an open-source, all-in-one business management application for freelancers, contractors, and solo entrepreneurs, offering invoici… | 74 | 15087 | active |
| microsoft/RD-Agent RD-Agent is a Microsoft open-source framework that uses LLM-powered agents to automate research and development processes such as factor mi… | 88 | 15014 | active |
| Tencent-Hunyuan/Hunyuan3D-2 Tencent's open-source Hunyuan3D-2 system for generating high-resolution 3D assets using large-scale diffusion models, supporting image-to-3… | 40 | 14994 | active |
| Botpress Botpress is an open-source platform and hub for building, deploying, and managing GPT/LLM-powered chatbots and AI agents, with a visual stu… | 67 | 14946 | active |
| livekit/agents LiveKit Agents is an open-source framework for building realtime, multimodal voice AI agents that run as programmable participants in LiveK… | 91 | 14870 | active |
| SesameAILabs/csm CSM (Conversational Speech Model) is Sesame's speech generation model that produces conversational audio from text and audio context, using… | 28 | 14728 | active |
| fathah/hermes-desktop Hermes One is a community-maintained native desktop app (Electron + React + TypeScript) for installing, configuring, and chatting with the … | 80 | 14407 | active |
| lsdefine/GenericAgent GenericAgent is a minimal self-evolving autonomous agent framework (~3K lines of seed code) that gives any LLM system-level control over a … | 80 | 14332 | active |
| Open-LLM-VTuber Open-LLM-VTuber is a cross-platform AI companion application that enables real-time voice conversations with LLMs, featuring a Live2D avata… | 62 | 13896 | active |
| BrowserOS BrowserOS is an open-source, Chromium-based agentic browser that lets AI agents (Claude Code, Codex, Cursor, MCP agents) drive a real brows… | 82 | 13886 | active |
| Fay Digital Human Framework Fay is an open-source Python digital human framework that connects 2.5D/3D/mobile/web digital humans or OpenAI-compatible LLMs to business … | 75 | 13549 | active |
| QwenLM/Qwen3-TTS Qwen3-TTS is a series of open-source text-to-speech models from Alibaba's Qwen team, supporting expressive and streaming speech generation,… | 45 | 13545 | active |
| doocs/md A clean, web-based Markdown editor that instantly renders Markdown into WeChat Official Account-compatible rich text, with custom themes, d… | 80 | 13412 | active |
| InsForge/InsForge InsForge is an open-source, all-in-one backend platform (BaaS) designed for AI coding agents, bundling Postgres, auth, storage, realtime, e… | 86 | 13129 | active |