Open-source llm-inference software

3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.

3358 products, primary matches first, then adoption-weighted; health v2 shown.

Product	Health v2	Stars	Maturity
Recursive Language Models (RLM) A plug-and-play Python inference library for Recursive Language Models (RLMs), a paradigm where an LLM programmatically examines and decomp…	76	5669	active
YILS-LIN/short-video-factory Short Video Factory is an open-source cross-platform desktop application that uses AI to generate and automatically edit short videos from …	76	5656	active
dotnet/skills A curated collection of agent skills and custom agents from the .NET team, packaged as plugins for AI coding agents like Copilot CLI and Cl…	63	5655	active
nextai-translator/bob-plugin-openai-translator A Bob plugin for macOS that uses OpenAI, Google Gemini, MiniMax, or OpenAI-compatible APIs to translate text, polish wording, and fix gramm…	93	5651	active
zarazhangrui/codebase-to-course A Claude Code skill that converts any codebase into a self-contained, interactive single-page HTML course for non-technical 'vibe coders'. …	45	5642	active
ConnectAI-E/feishu-openai A self-hosted Go application that integrates OpenAI models (GPT-4, GPT-4V, DALL·E-3, Whisper) into Feishu/Lark as a chatbot. It supports vo…	32	5633	active
HisMax/RedInk RedInk is a self-hostable web application that generates complete Xiaohongshu (Little Red Book) image-and-text posts from a single topic se…	54	5632	active
lm-sys/RouteLLM RouteLLM is a Python framework for serving and evaluating LLM routers that route queries between strong and weak models to cut costs. It pr…	25	5626	active
maziyarpanahi/openmed OpenMed is a local-first healthcare AI SDK for clinical named-entity recognition and HIPAA PII/PHI de-identification that runs entirely on-…	84	5623	active
sohzm/cheating-daddy Cheating Daddy is a free, open-source Electron desktop app that acts as a real-time AI assistant during video calls, interviews, and meetin…	72	5616	active
workweave/router Workweave Router is a model router for agentic AI systems that routes each prompt to the most cost-effective model capable of handling it, …	66	5596	active
MiniGPT-4 Official code for MiniGPT-4 and MiniGPT-v2, vision-language models that align a frozen visual encoder with a frozen LLM (Vicuna) via a sing…	30	25605	maintenance
karpathy/llm-council A local web app that sends a user's question to multiple LLMs via OpenRouter, has them anonymously review and rank each other's answers, an…	37	25563	maintenance
ngxson/smolvlm-realtime-webcam A browser-based demo that streams webcam frames to a llama.cpp server running SmolVLM 500M for real-time object detection and scene descrip…	26	5577	active
ATH-MaaS/ComfyUI-Copilot ComfyUI-Copilot is an AI-powered custom node for ComfyUI that acts as an intelligent assistant for building, debugging, and optimizing imag…	61	5539	active
father-bot/chatgpt_telegram_bot A self-hostable Telegram bot that brings ChatGPT and Claude models into Telegram using your own OpenAI/Anthropic/OpenRouter API keys. It su…	73	5538	active
ysharma3501/LuxTTS LuxTTS is a lightweight zipvoice-based text-to-speech model for high-quality zero-shot voice cloning, generating clear 48kHz speech at up t…	51	5433	active
docker/genai-stack A Docker Compose-based starter stack for building GenAI applications, combining LangChain, Ollama for local LLM inference, and Neo4j as a d…	70	5416	active
EverMind-AI/Raven Raven is an open-source, self-improving agent harness built on EverOS that combines terminal-first execution, local tracing, long-term memo…	81	5409	active
TaskingAI/TaskingAI TaskingAI is an open-source Backend-as-a-Service (BaaS) platform for developing and deploying LLM-based AI agents. It unifies access to hun…	17	5406	active
VibeSDK An open-source SDK for building your own vibe-coding platform where users describe full-stack apps and an AI coding agent plans, edits, dep…	71	5401	active
lingodotdev/lingo.dev Lingo.dev is an open-source localization engineering toolkit that turns LLMs into stateful translation APIs, persisting glossaries, brand v…	88	5398	active
NVIDIA/Model-Optimizer NVIDIA Model Optimizer (ModelOpt) is a Python library of state-of-the-art model optimization techniques including quantization, pruning, di…	92	5386	active
deepseek-ai/DeepSeek-VL2 DeepSeek-VL2 is a series of Mixture-of-Experts vision-language models (Tiny, Small, and 4.5B activated parameters) with inference code and …	22	5376	active
looplj/axonhub AxonHub is an open-source AI gateway written in Go that lets any SDK call 100+ LLMs through a unified API. It provides built-in failover, l…	81	5375	active
lmstudio-ai/lms lms is the official command-line tool for LM Studio, built on the lmstudio.js SDK. It lets users manage local LLMs (list, download, load, u…	70	5368	active
winfunc/deepreasoning DeepReasoning is a high-performance Rust-based LLM inference API and chat UI that combines DeepSeek R1's chain-of-thought reasoning traces …	28	5361	active
Roo Code Roo Code is a VS Code extension that brings a team of AI agents into the editor to assist with coding, design, debugging, and documentation…	10	24296	maintenance
aipoch/open-science Open Science is an open-source, local-first, model-agnostic AI research workbench for reproducible scientific research. It combines a coord…	77	5303	active
BuilderIO/ai-shell AI Shell is an open-source CLI tool that converts natural language prompts into shell commands using OpenAI, with options to run, revise, o…	54	5301	active
neo4j-labs/llm-graph-builder A web application that transforms unstructured data (PDFs, DOCs, TXTs, YouTube videos, web pages) into a knowledge graph stored in Neo4j us…	88	5264	active
Anil-matcha/AI-Youtube-Shorts-Generator An open-source Python tool that turns long-form YouTube videos into vertical 9:16 short clips using LLM-based highlight detection, Whisper …	68	5262	active
camel-ai/oasis OASIS is an open-source social media simulator that uses large language model and rule-based agents to mimic the behavior of up to one mill…	72	5245	active
su-kaka/gcli2api A self-hosted Python service that converts GeminiCLI and Antigravity into OpenAI-, Gemini-, and Claude-compatible API endpoints. It can be …	63	5243	active
DeepResearch A TypeScript agent that iteratively searches the web, reads webpages, and reasons with an LLM (Gemini, OpenAI, or local) until it finds a c…	55	5237	active
pollinations/pollinations Pollinations is an open-source generative AI platform offering free APIs for text, image, and vision model inference without requiring API …	77	5226	active
Kode AI Agent Platform Learn Claude Code is a Python library that provides a minimal harness for building Claude-like AI agents. It includes CLI tools and tutoria…	60	5225	active
Adam-CAD/CADAM CADAM is an open-source web application that converts natural language descriptions and images into 3D CAD models using LLM-powered agents …	73	5210	active
linuxhsj/openclaw-zero-token OpenClaw Zero Token is a fork of OpenClaw that provides a unified gateway to major LLM providers (ChatGPT, Claude, Gemini, DeepSeek, Qwen, …	71	5206	active
kodu-ai/claude-coder Kodu (Claude Coder) is an autonomous AI coding agent distributed as a VS Code extension. It takes natural-language task descriptions and bu…	28	5205	active
Transformer Lab Transformer Lab is an open-source desktop application (built with Electron and Python) that provides a unified GUI for training, fine-tunin…	82	5195	active
AgnesAI-Labs/AgnesAI-Models Official gateway and model catalog for Agnes AI, providing OpenAI-compatible API access to multimodal foundation models covering text, imag…	55	5192	active
vllm-project/aibrix AIBrix is an open-source, cloud-native framework from the vLLM project providing essential building blocks for scalable, cost-efficient Gen…	88	5149	active
Codeium/Windsurf windsurf.vim is a Vim and Neovim plugin providing free, AI-powered code autocomplete as a Copilot alternative. It integrates the Windsurf (…	62	5142	active
OpenBMB/AgentVerse AgentVerse is a Python framework for deploying multiple LLM-based agents, offering two frameworks: task-solving (multi-agent collaboration …	21	5141	active
icip-cas/PPTAgent PPTAgent is an agentic Python framework that automatically generates PowerPoint presentations from documents using a two-stage, edit-based …	79	5122	active
tencentmusic/supersonic SuperSonic is an AI+BI platform that combines Chat BI (natural language querying powered by LLMs) with Headless BI (a governed semantic lay…	62	5112	active
ztxz16/fastllm fastllm is a high-performance C++ LLM inference library with its own custom operators, requiring no PyTorch dependency. It supports tensor-…	75	5110	active
pytorch/executorch ExecuTorch is PyTorch's framework for exporting and running AI models on-device across mobile, embedded, and edge hardware, with a tiny (~5…	94	5101	active
manycore-research/SpatialLM SpatialLM is a 3D large language model that processes point cloud data (from monocular video, RGBD images, or LiDAR) and generates structur…	59	5094	active
huggingface/text-embeddings-inference Text Embeddings Inference (TEI) is a Rust-based toolkit from Hugging Face for deploying and serving open-source text embedding, re-ranker, …	93	5079	active
deepseek-ai/DeepSeek-V2 DeepSeek-V2 is a strong, economical Mixture-of-Experts language model released with open weights, inference code, and evaluation tooling. T…	25	5042	active
unicity-aos/capsule-react A Rust ReAct loop coordinator that implements a stateless state machine for the reasoning-and-action cycle in agent systems. It is part of …	70	5041	active
developersdigest/llm-answer-engine A Perplexity-inspired AI answer engine built with Next.js, the Vercel AI SDK, LangChain.JS, Groq/Mixtral, Brave Search, Serper, and OpenAI.…	58	5035	active
Gaia node GaiaNet node is a shell-based installer and runtime for running your own decentralized AI agent service locally. It deploys an LLM inferenc…	45	5015	active
AG2 AG2 (formerly AutoGen) is an open-source Python framework for building AI agents and orchestrating multi-agent systems, supporting LLM inte…	85	5003	active
pkuliyi2015/multidiffusion-upscaler-for-automatic1111 An Automatic1111 Stable Diffusion WebUI extension that enables generating and upscaling ultra-large images (2K+) on GPUs with limited VRAM …	31	4995	active
gd3kr/BlenderGPT A Blender add-on that integrates OpenAI's GPT-4/GPT-3.5 into Blender's UI, letting users control the 3D software with natural language comm…	31	4987	active
mistralai/mistral-vibe Mistral Vibe is an open-source command-line coding agent powered by Mistral's models, providing a conversational interface to explore and m…	77	4986	active
going-doer/Paper2Code PaperCoder is a multi-agent LLM system that automatically generates a code repository from a machine learning research paper via a planning…	49	4982	active
EvoScientist EvoScientist is a Python-based multi-agent framework for building self-evolving AI scientists that plan, research, code, analyze data, and …	83	4980	active
joonspk-research/generative_agents The official code release for the Stanford 'Generative Agents' research paper, implementing computational agents powered by LLMs that simul…	29	22237	maintenance
elie222/rakazo Rakazo is an open-source, self-hosted platform for running persistent AI teammates (bots) that can use a browser, terminal, and desktop, re…	81	4947	active
google-deepmind/alphageometry Google DeepMind's implementation of AlphaGeometry and DDAR, AI systems that solve Olympiad-level geometry theorem proving problems. It comb…	52	4899	active
Martian-Engineering/lossless-claw A Lossless Context Management (LCM) plugin for OpenClaw that replaces sliding-window context truncation with a DAG-based summarization syst…	79	4893	active
llm-d llm-d is a Kubernetes-native distributed LLM inference serving stack that orchestrates model servers like vLLM and SGLang across clusters. …	83	4891	active
Deep Java Library (DJL) Deep Java Library (DJL) is an engine-agnostic, high-level deep learning framework for Java that supports backends like PyTorch, TensorFlow,…	96	4857	active
chaitin/MonkeyCode MonkeyCode is an open-source, enterprise-grade AI coding platform that combines managed development environments, AI model management, and …	78	4811	active
SakuraLLM/SakuraLLM SakuraLLM is a series of fine-tuned open large language models for Japanese-to-Chinese translation specialized in light novel and Galgame (…	58	4785	active
Integuru-AI/Integuru Integuru is an AI agent that reverse-engineers platforms' internal APIs by analyzing browser network requests (HAR files) and building depe…	59	4781	active
facebookresearch/lingua Meta Lingua is a minimal, fast LLM training and inference library built on easy-to-modify PyTorch components for research purposes. It supp…	33	4771	active
deepseek-ai/Engram Official implementation of Engram, a conditional memory module from DeepSeek that modernizes N-gram embeddings for O(1) lookup as a new spa…	41	4735	active
GeekAI GeekAI is an open-source, self-hosted AI content creation platform integrating chat, image generation (MidJourney, DALL-E, Stable Diffusion…	92	4709	active
Rivet Rivet is an open-source visual AI programming environment consisting of a desktop IDE for building AI agents and prompt chains via a node-b…	73	4708	active
u14app/deep-research Deep Research is a web application that uses any large language model (Gemini, OpenAI, Anthropic, DeepSeek, Grok, Ollama, etc.) combined wi…	66	4691	active
OpenNMT/CTranslate2 CTranslate2 is a C++ and Python library for fast, memory-efficient inference of Transformer models on CPU and GPU. It uses quantization, la…	94	4684	stable
OpenMinis OpenMinis is a free, open-source mobile AI agent app for iOS and Android that connects leading LLM providers (Claude, GPT, Gemini, etc.) vi…	76	4673	active
Tencent/TNN TNN is a high-performance, lightweight deep learning inference framework developed by Tencent Youtu Lab, supporting mobile, desktop, and se…	28	4656	active
ruc-datalab/DeepAnalyze DeepAnalyze is the first agentic LLM (DeepAnalyze-8B, built on Qwen) for autonomous data science, capable of running the full pipeline from…	61	4655	active
promptslab/Promptify Promptify is a Python library that turns LLMs into task-based NLP engines, providing high-level APIs for NER, classification, QA, and custo…	62	4636	active
TencentCloudADP/youtu-agent Youtu-Agent is a Python framework for building, running, and evaluating autonomous LLM agents, supporting ReAct-style single agents and pla…	52	4622	active
JetBrains/koog Koog is a JetBrains-built Kotlin/Java framework for creating fault-tolerant, enterprise-ready AI agents on the JVM, with multiplatform depl…	86	4617	active
hao-ai-lab/FastVideo FastVideo is a unified Python framework for post-training and real-time inference of video diffusion models, covering data preprocessing, f…	79	4616	active
mini-swe-agent SWE-agent is a Python framework that lets a language model autonomously use tools to fix GitHub issues, find cybersecurity vulnerabilities,…	67	20450	maintenance
VRSEN/agency-swarm Agency Swarm is a Python framework for building multi-agent AI applications on top of the OpenAI Agents SDK. It lets developers define spec…	91	4601	active
FareedKhan-dev/all-agentic-architectures A Python library packaging 35 production-grade agentic AI architectures (Reflexion, LATS, GraphRAG, MemGPT, Voyager, etc.) as runnable Arch…	57	4600	active
get-convex/chef Chef is an AI-powered full-stack web app builder by the Convex team, forked from bolt.diy, that generates apps with a built-in reactive dat…	55	4599	active
kyegomez/tree-of-thoughts A Python library implementing the Tree of Thoughts algorithm for deliberate, multi-step problem solving with large language models. It prov…	64	4595	active
MemeCalculate/moyin-creator Moyin Creator is a production-grade AI film creation desktop application that covers the full pipeline from script to finished video, with …	73	4580	active
fixie-ai/ultravox Ultravox is a fast multimodal LLM that understands human speech directly without a separate ASR stage, by projecting audio into the LLM's e…	40	4577	active
phodal/auto-dev AutoDev is an AI-native multi-agent coding development platform built on Kotlin Multiplatform, distributed as an IntelliJ IDEA plugin, VS C…	80	4548	active
0xNyk/council-of-high-intelligence A shell-based skill/plugin that runs structured multi-persona deliberation for hard decisions across AI coding agents like Claude Code, Cod…	72	4539	active
OAID/Tengine Tengine is a lightweight, high-performance, modular deep learning inference engine developed by OPEN AI LAB for embedded and edge devices. …	24	4531	active
kaixindelele/ChatPaper ChatPaper is a Python tool that uses ChatGPT to summarize arXiv papers, translate full PDFs, polish academic writing, and simulate peer rev…	59	19943	maintenance
Embabel Agent Framework Embabel Agent is an agentic AI framework for the JVM, written in Kotlin with first-class Java and Spring Boot support. It provides typed, g…	87	4513	active
openai/harmony OpenAI's Rust library (with first-class Python bindings) for rendering and parsing the harmony response format used by its open-weight gpt-…	49	4506	active
microsoft/LMOps A Microsoft research initiative collecting code and papers on fundamental techniques for building AI products with large language models, c…	75	4475	active
crmne/ruby_llm RubyLLM is a Ruby framework providing a unified, expressive interface to all major AI providers (OpenAI, Anthropic, Google, Ollama, and any…	88	4472	active
anc95/ChatGPT-CodeReview A GitHub App and GitHub Action that automatically reviews pull requests using ChatGPT/OpenAI models, posting AI-generated review comments o…	89	4466	active
EvolvingLMMs-Lab/lmms-eval lmms-eval is a unified Python framework for evaluating large multimodal models across text, image, video, and audio tasks, with 100+ benchm…	87	4453	active