Open-source llm-inference software
3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
3358 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| Outlines Outlines is a Python library for structured generation that guarantees LLM outputs conform to a desired structure (JSON schemas, regex, gra… | 93 | 15920 | active |
| GeeeekExplorer/nano-vllm A lightweight vLLM-style LLM inference engine implemented from scratch in about 1,200 lines of Python. It offers fast offline inference wit… | 51 | 15834 | active |
| modelscope/ms-swift ms-swift is a ModelScope community framework for fine-tuning, inference, evaluation, quantization, and deployment of 600+ text LLMs and 400… | 90 | 15809 | active |
| Anionex/banana-slides Banana Slides is an AI-native presentation generator built on the nano banana pro image model that turns ideas, outlines, documents, or tem… | 66 | 15694 | active |
| plandex-ai/plandex Plandex is an open-source, terminal-based AI coding agent that plans and executes large, multi-step coding tasks across many files in real-… | 46 | 15656 | active |
| ggml-org/ggml ggml is a plain C/C++ tensor library for machine learning with no dependencies, designed to be simple, portable, and efficient. It supports… | 99 | 15480 | stable |
| NVIDIA/TensorRT-LLM NVIDIA's TensorRT-LLM is a Python library and runtime for defining and serving Large Language Models with state-of-the-art inference optimi… | 83 | 14851 | active |
| llmware-ai/llmware llmware is a Python framework for building enterprise RAG pipelines and LLM-based applications using small, specialized open-source models.… | 90 | 14816 | active |
| RWKV RWKV is a novel language model architecture that combines RNN efficiency (linear time, constant memory, no KV-cache) with transformer-level… | 66 | 14744 | active |
| AI Scientist The AI Scientist is a system that uses large language models to autonomously perform end-to-end scientific research, from idea generation a… | 46 | 14620 | active |
| aiming-lab/AutoResearchClaw AutoResearchClaw is a Python framework for fully autonomous, self-evolving research that takes an idea from chat to a generated paper using… | 70 | 14582 | active |
| FreeToken FreeToken is an edge-native Mixture-of-Experts (MoE) LLM serving engine that runs frontier-scale open-weight models on consumer hardware by… | 79 | 14561 | active |
| EleutherAI/lm-evaluation-harness A Python framework for few-shot evaluation of language models across hundreds of standard benchmarks (HellaSwag, MMLU, BIG-Bench, etc.). It… | 91 | 14198 | active |
| netease-youdao/QAnything QAnything is a local knowledge base question-answering system by NetEase Youdao that supports uploading and querying various document forma… | 61 | 14188 | active |
| FujiwaraChoki/MoneyPrinter MoneyPrinter is a Python application that automates the creation of YouTube Shorts from a user-provided video topic, using local Ollama mod… | 56 | 14059 | active |
| Microsoft Agent Framework Microsoft Agent Framework (MAF) is an open-source, multi-language framework for building, orchestrating, and deploying production-grade AI … | 82 | 13999 | active |
| Physical-Intelligence/openpi Open-source repository from Physical Intelligence containing vision-language-action (VLA) models for robotics, including π₀, π₀-FAST, and π… | 64 | 13991 | active |
| Instructor Instructor is a Python library that patches LLM clients to return validated, type-safe structured data using Pydantic models, with automati… | 94 | 13980 | stable |
| apache/tvm Apache TVM is an open machine learning compiler framework that takes pre-trained models and compiles them into optimized, deployable module… | 95 | 13840 | active |
| nanobrowser/nanobrowser Nanobrowser is an open-source Chrome extension that provides AI-powered web automation through a multi-agent system (Planner, Navigator, et… | 67 | 13818 | active |
| Omi Omi is an open-source AI wearable and companion app ecosystem (necklace pendant, smart glasses, desktop and mobile apps) that captures conv… | 89 | 13763 | active |
| Lightning-AI/litgpt LitGPT is a Python library providing from-scratch, hackable implementations of 20+ open-source large language models with recipes for pretr… | 87 | 13718 | active |
| miuuyy/codex-chatgpt-web A local bridge application that exposes ChatGPT Web (including Pro tiers) as a native model inside the OpenAI Codex app, routing model turn… | 80 | 13664 | active |
| CodebuffAI/freebuff Freebuff is a free AI coding agent available as a desktop app, CLI, web builder, cloud IDE, and chat interface, funded by text ads instead … | 73 | 13614 | active |
| XiaomiMiMo/MiMo-Code MiMoCode is a terminal-native AI coding assistant that can read and write code, run commands, manage Git, and maintain persistent project m… | 79 | 13584 | active |
| Cactus Needle 2 is a 45M-parameter open language model packaged as a single 14MB binary for tool calling, device use, and structured extraction on… | 60 | 13581 | active |
| nidhinjs/prompt-master A Claude skill (plugin) that generates and refines precise prompts for a wide range of AI tools like ChatGPT, Cursor, Midjourney, and Stabl… | 56 | 13535 | active |
| huggingface/speech-to-speech A low-latency, fully modular voice-agent pipeline (VAD -> STT -> LLM -> TTS) built from open-source models, exposed via the OpenAI Realtime… | 85 | 13485 | active |
| The-PR-Agent/pr-agent PR-Agent is an open-source, AI-powered code review agent that automatically reviews pull requests, generates descriptions, suggests improve… | 97 | 13427 | active |
| NVIDIA/TensorRT NVIDIA TensorRT is an SDK for high-performance deep learning inference on NVIDIA GPUs, comprising an inference compiler that converts train… | 93 | 13362 | stable |
| cloudwego/eino Eino is a Go-based framework for building LLM and AI applications, inspired by LangChain and LlamaIndex but following Go conventions. It pr… | 84 | 13269 | active |
| LangChain4j LangChain4j is an open-source Java library for building LLM-powered applications on the JVM, providing a unified API over 20+ LLM providers… | 97 | 13266 | active |
| Portkey AI Gateway Portkey AI Gateway is an open-source, TypeScript-based AI gateway that routes requests to 1,600+ LLMs across many providers through a singl… | 71 | 13079 | active |
| lllyasviel/stable-diffusion-webui-forge A fork/platform built on top of AUTOMATIC1111's Stable Diffusion WebUI that optimizes resource management, speeds up inference, and adds ex… | 29 | 13071 | active |
| ACE-Step ACE-Step 1.5 is an open-source music generation foundation model combining a language model planner with a Diffusion Transformer to create … | 77 | 13070 | active |
| deepseek-ai/FlashMLA FlashMLA is DeepSeek's library of optimized CUDA attention kernels implementing Multi-head Latent Attention (MLA), including dense and toke… | 64 | 13063 | active |
| PaddlePaddle/PaddleNLP PaddleNLP is an easy-to-use NLP and large language model development kit built on the PaddlePaddle deep learning framework, with a large pr… | 59 | 12977 | active |
| NoFxAiOS/nofx NOFX is an open-source, self-hosted AI trading terminal where a language model acts as the trading strategy, running a continuous decide-an… | 59 | 12945 | active |
| Pocket Flow An AI agent built with the Pocket Flow LLM framework that crawls GitHub repositories, analyzes codebases to identify core abstractions, and… | 55 | 12683 | active |
| elie222/inbox-zero Inbox Zero is an open-source AI email assistant web application that organizes your inbox, pre-drafts replies in your voice, blocks cold em… | 97 | 12645 | active |
| LLM A CLI tool and Python library for running prompts against large language models from OpenAI, Anthropic, Google, and many other providers, i… | 95 | 12559 | stable |
| BentoML OpenLLM is a Python framework and CLI for self-hosting open-source large language models (Llama, DeepSeek, Qwen, Mistral, etc.) as OpenAI-c… | 76 | 12552 | active |
| TheR1D/shell_gpt ShellGPT (sgpt) is a Python command-line productivity tool that uses LLMs like GPT-4/5 or local models via Ollama to generate shell command… | 71 | 12291 | active |
| getomni-ai/zerox Zerox is a library (Node.js and Python packages) that performs OCR and document extraction by converting files like PDFs, DOCX, and images … | 32 | 12249 | active |
| bytedance/trae-agent Trae Agent is an LLM-based CLI agent from ByteDance for general-purpose software engineering tasks, executing workflows via tools like file… | 45 | 12135 | active |
| LMCache/LMCache LMCache is a KV cache management layer for LLM inference that stores, compresses, and reuses KV caches across requests, sessions, and servi… | 88 | 12133 | active |
| altic-dev/FluidVoice FluidVoice is an open-source (GPLv3) macOS dictation app that transcribes speech to text entirely on-device using models like Parakeet, Whi… | 83 | 11931 | active |
| jd-opensource/joyagent-jdgenie JoyAgent-JDGenie is an open-source, end-to-end, production-grade general-purpose multi-agent product built in Java. It directly answers use… | 36 | 11931 | active |
| NVIDIA/cosmos NVIDIA Cosmos is an open platform of omnimodal world foundation models, datasets, and tools for building Physical AI systems such as robots… | 69 | 11892 | active |
| Arize Phoenix Arize Phoenix is an open-source AI observability and evaluation platform for tracing, debugging, and evaluating LLM applications. It suppor… | 95 | 11875 | active |
| BeehiveInnovations/pal-mcp-server PAL MCP (formerly Zen MCP) is a Model Context Protocol server that lets AI coding CLIs like Claude Code, Gemini CLI, and Codex CLI orchestr… | 44 | 11773 | active |
| tensorzero/tensorzero TensorZero is an open-source LLMOps platform that unifies an LLM gateway, observability, evaluation, optimization, and experimentation behi… | 10 | 11716 | active |
| microsoft/autogen AutoGen is a Python framework from Microsoft for building multi-agent AI applications where LLM-powered agents can act autonomously or coll… | 62 | 61080 | maintenance |
| github/copilot.vim A Vim/Neovim plugin that integrates GitHub Copilot, an AI pair programmer that generates code suggestions from natural language prompts. Su… | 63 | 11695 | active |
| elder-plinius/G0DM0D3 G0DM0D3 is an open-source, multi-model AI chat interface designed for red-teaming, cognition research, and unrestricted LLM interaction. It… | 53 | 11643 | active |
| yizhiyanhua-ai/fireworks-tech-graph A Python-based Agent Skill for Codex and Claude Code that converts natural language descriptions (English or Chinese) into production-quali… | 75 | 11601 | active |
| linyqh/NarratoAI NarratoAI is an open-source, self-hosted AI-powered video commentary and automated editing tool. It uses large language models to generate … | 86 | 11474 | active |
| wangrongding/wechat-bot A Node.js-based multi-platform instant messaging AI agent built on Wechaty that connects WeChat, Telegram, WhatsApp, and Lark to LLM provid… | 65 | 11421 | active |
| kyutai-labs/moshi Moshi is a speech-text foundation model and full-duplex spoken dialogue framework from Kyutai, built around the Mimi streaming neural audio… | 56 | 11263 | active |
| github/copilot-cli GitHub Copilot CLI brings the Copilot coding agent to the terminal, letting developers build, debug, and refactor code through natural lang… | 95 | 11253 | active |
| MiniCPM MiniCPM is a family of small, state-of-the-art on-device language models from OpenBMB, with MiniCPM5-1B being a dense 1B Transformer for lo… | 74 | 11232 | active |
| MiniCPM MiniCPM is a family of compact multimodal large language models designed for efficient image, video, and audio understanding on mobile devi… | 74 | 11232 | active |
| openai/openai-node The official JavaScript/TypeScript SDK for the OpenAI REST API, auto-generated from OpenAI's OpenAPI specification. It provides typed acces… | 96 | 11225 | stable |
| Doriandarko/claude-engineer An interactive AI assistant framework built on Anthropic's Claude 3.5 Sonnet that can autonomously generate and manage its own tools during… | 24 | 11211 | active |
| LTX-Video Official Python package from Lightricks providing inference pipelines and LoRA training for LTX-2/LTX-2.5, an open-weights DiT-based founda… | 45 | 11138 | active |
| JOYCEQL/magic-resume Magic Resume is a free online AI-powered resume editor built with TanStack Start, TypeScript, and Framer Motion. It offers real-time previe… | 91 | 11103 | active |
| Triton Inference Server NVIDIA Triton Inference Server is an open-source inference serving software that deploys AI models from multiple frameworks (TensorRT, PyTo… | 98 | 11075 | stable |
| langchain-ai/open-swe Open SWE is an open-source framework from LangChain for building an organization's internal asynchronous coding agent, built on LangGraph a… | 87 | 11049 | active |
| SparkAudio/Spark-TTS Spark-TTS is an LLM-based text-to-speech system built on Qwen2.5 that generates speech directly from single-stream decoupled speech tokens.… | 24 | 10998 | active |
| huggingface/chat-ui Chat UI is the open source SvelteKit web application powering HuggingChat, providing a browser-based chat interface for LLMs. It connects t… | 79 | 10987 | active |
| NoeFabris/opencode-antigravity-auth An OpenCode plugin that authenticates against Google's Antigravity IDE via OAuth, letting users access Gemini and Claude models with their … | 10 | 10983 | active |
| alibaba/spring-ai-alibaba Spring AI Alibaba is a production-ready agentic AI framework for Java developers, built on Spring AI and a DAG-based Graph runtime. It prov… | 77 | 10966 | active |
| Thysrael/Horizon Horizon is an AI-powered news aggregation platform that fetches content from sources like Hacker News, GitHub, RSS, Reddit, and Telegram, s… | 61 | 10909 | active |
| multimodal-art-projection/YuE YuE is a family of open-source foundation models based on the LLaMA2 architecture that generate full songs (up to five minutes) with vocals… | 87 | 10822 | active |
| WasmEdge WasmEdge is a lightweight, high-performance, extensible WebAssembly runtime hosted by CNCF, designed for cloud native, edge, and decentrali… | 98 | 10811 | active |
| huggingface/ml-intern An open-source autonomous ML engineer agent from Hugging Face that reads papers, writes code, trains models, and ships ML work using the Hu… | 10 | 10810 | active |
| sashabaranov/go-openai An unofficial Go client library for the OpenAI API covering chat completions, the Responses API, embeddings, image generation, audio transc… | 96 | 10778 | active |
| presenton/presenton Presenton is an open-source AI presentation generator and API that creates editable PPTX and PDF slide decks from prompts, documents, or cu… | 80 | 10756 | active |
| SkyPilot SkyPilot is an open-source AI compute platform that unifies fragmented infrastructure (Kubernetes, Slurm, VMs, 20+ clouds) into a single po… | 100 | 10724 | active |
| AntonOsika/gpt-engineer gpt-engineer is a Python CLI platform for experimenting with AI code generation, letting users specify software in natural language and wat… | 10 | 55049 | maintenance |
| nexmoe/VidBee VidBee is a free, open-source Electron desktop app for downloading video and audio from 1000+ sites (via yt-dlp) and importing local media.… | 80 | 10689 | active |
| Acly/krita-ai-diffusion A Krita plugin providing a streamlined interface for AI image generation, inpainting, and outpainting within the Krita painting application… | 92 | 10668 | active |
| chathub-dev/chathub ChatHub is an all-in-one AI chat client, originally a Chrome/Edge browser extension and now also a web app and mobile/desktop apps, that le… | 58 | 10658 | active |
| abetlen/llama-cpp-python Python bindings for llama.cpp providing low-level ctypes access to the C API plus a high-level API for text completion with OpenAI-like int… | 92 | 10638 | active |
| bigscience-workshop/petals Petals is a Python library that lets you run and fine-tune large language models (Llama 3.1, Mixtral, Falcon, BLOOM) on a BitTorrent-style … | 23 | 10614 | active |
| a16z-infra/ai-town AI Town is a deployable starter kit for building a virtual town where AI characters live, chat, and socialize, inspired by the Generative A… | 70 | 10589 | active |
| NVIDIA/personaplex PersonaPlex is a real-time, full-duplex speech-to-speech conversational model from NVIDIA that supports persona control via text role promp… | 44 | 10567 | active |
| GitHub Copilot SDK A multi-platform SDK exposing the GitHub Copilot CLI agent runtime for programmatic use, with official SDKs for Python, TypeScript, Go, .NE… | 79 | 10557 | active |
| akitaonrails/ai-memory A Rust CLI and MCP server that provides long-term memory for AI coding agent CLIs like Claude Code, Codex, and Devin. It captures session c… | 78 | 10555 | active |
| Easy Diffusion Easy Diffusion is a 1-click installer and browser-based UI for running Stable Diffusion text-to-image generation locally on your PC. It bun… | 76 | 10473 | active |
| RunAnywhere RunAnywhere is a set of cross-platform SDKs (Swift, Kotlin, React Native, Flutter, TypeScript, C++) over a shared C++ core for running AI m… | 86 | 10332 | active |
| deepseek-ai/DeepEP DeepEP is a high-performance GPU communication library for expert parallelism (EP) in MoE training and inference, providing high-throughput… | 55 | 10263 | active |
| xai-org/grok-1 xAI's open release of the Grok-1 open-weights model (314B-parameter Mixture-of-Experts LLM) with JAX example code for loading and running i… | 26 | 52232 | maintenance |
| Narcooo/inkos InkOS is an AI agent system for story creation and multilingual translation, supporting long-form novels, scripts, interactive games, and I… | 79 | 10222 | active |
| InternVL InternVL is a family of open-source multimodal large language models (vision-language models) that combine vision transformers with LLMs to… | 34 | 10163 | active |
| open-jarvis/OpenJarvis OpenJarvis is a Python framework for building local-first personal AI agents that run on your own hardware, with cloud APIs as an optional … | 70 | 10144 | active |
| Moondream Moondream is an open-weight family of small, efficient vision language models (2B to 9B MoE) that perform image captioning, visual question… | 58 | 10127 | active |
| mengxi-ream/read-frog Read Frog is an open-source AI-powered browser extension for language learning and immersive translation. It translates web pages, selected… | 84 | 10112 | active |
| yihong0618/bilingual_book_maker A Python CLI tool that uses AI translation services (ChatGPT, Claude, Gemini, DeepL, and others) to create bilingual versions of epub, txt,… | 96 | 9825 | active |
| Tiiny-AI/PowerInfer PowerInfer is a CPU/GPU hybrid LLM inference engine that exploits activation locality to run large language models fast on consumer-grade h… | 60 | 9814 | active |