Open-source llm-inference software

3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.

3358 products, primary matches first, then adoption-weighted; health v2 shown.

Product	Health v2	Stars	Maturity
OpenClaw OpenClaw is an open‑source AI assistant that runs on your own hardware and connects to dozens of chat platforms. It provides a gateway, plu…	80	392291	active
OpenCode OpenCode is an open source AI coding agent that helps developers write code from the terminal, desktop app, or IDE extension. It supports 7…	83	215433	active
AutoGPT AutoGPT is an open-source platform for building, deploying, and running AI agents that execute complete workflows, via natural-language cha…	94	187630	active
Ollama Ollama is a local server and CLI tool for running and managing open large language models. It provides a REST API, integrations with coding…	93	183649	active
Transformers Hugging Face Transformers is a Python library that serves as the model-definition framework for state-of-the-art machine learning models ac…	95	167798	stable
langflow-ai/langflow Langflow is an open‑source Python‑based platform that provides a visual low‑code editor for building, testing, and deploying AI agents and …	95	155852	active
Open WebUI Open WebUI is a self‑hosted, web‑based interface for interacting with large language models. It supports local Ollama models, OpenAI‑compat…	88	152948	active
LangChain LangChain is an open-source Python library for building LLM-powered applications and autonomous agents. It provides composable components, …	95	148099	active
LangChain LangChain is an open-source Python framework for building LLM-powered agents and applications, providing a configurable agent harness (crea…	95	148099	stable
harry0703/MoneyPrinterTurbo MoneyPrinterTurbo is an AI-powered application that generates high-definition short videos from a topic or keyword. It automatically writes…	92	131685	active
ggml-org/llama.cpp llama.cpp is a high-performance C/C++ library for running large language models locally. It provides both a library API and ready-to-use co…	93	131193	active
nextlevelbuilder/ui-ux-pro-max-skill UI UX Pro Max is an AI‑driven CLI that offers design intelligence for creating professional user interfaces across many frameworks. It prov…	79	129906	active
openai/codex OpenAI's Codex CLI is a lightweight coding agent that runs locally in your terminal, helping you write, review, and refactor code with AI a…	83	129656	active
Pi Pi is a TypeScript toolkit that provides a unified multi‑provider LLM API, an agent runtime with tool calling, and a terminal‑based coding …	82	115982	active
Pi Pi is a TypeScript AI agent toolkit providing a unified multi-provider LLM API, an agent runtime with tool calling, a terminal UI library, …	82	115982	active
TradingAgents TradingAgents is a Python framework that enables multi‑agent LLM‑driven financial trading strategies. It provides data access contracts, mo…	87	109273	active
TradingAgents TradingAgents is a Python framework that orchestrates multiple LLM-powered agents to analyze financial markets and make trading decisions. …	87	109273	active
deepseek-ai/DeepSeek-V3 Official repository for DeepSeek-V3, a 671B-parameter Mixture-of-Experts language model (37B activated per token) with open weights, infere…	26	104498	active
vLLM vLLM is a Python library that provides fast, high‑throughput inference and serving for large language models. It supports advanced features…	98	94014	active
vLLM vLLM is a high-throughput, memory-efficient inference and serving engine for large language models, featuring PagedAttention, continuous ba…	98	94014	active
deepseek-ai/DeepSeek-R1 DeepSeek-R1 is a family of open-weight large language models trained with large-scale reinforcement learning for reasoning, including DeepS…	21	91900	active
NextChat NextChat is an open‑source, cross‑platform AI chat assistant that lets users interact with many LLMs (GPT‑5, Claude, Gemini, etc.) without …	73	88834	active
NextChat NextChat is a lightweight, fast open-source AI chat assistant web UI supporting multiple LLM providers like OpenAI GPT-4, Claude, Gemini, D…	73	88834	active
abi/screenshot-to-code An AI-powered application that converts screenshots, mockups, Figma designs, and screen recordings into clean functional code (HTML/Tailwin…	71	80005	active
OmniRoute OmniRoute is a free, MIT-licensed AI gateway that exposes a single OpenAI-compatible endpoint routing to 350+ LLM providers (90+ with free …	78	77757	active
GPT4All GPT4All is an open‑source desktop application that runs large language models locally on Windows, macOS and Linux. It provides a private AI…	39	77391	active
hiyouga/LlamaFactory LlamaFactory is an open‑source Python platform that streamlines fine‑tuning, quantization, and inference of over 100 LLM and VLM models. It…	81	75500	active
binary-husky/gpt_academic GPT Academic is a self-hosted Python application providing a practical chat interface for LLMs like GPT, GLM, Qwen, DeepSeek, and Claude, w…	47	71417	active
FoundationAgents/MetaGPT MetaGPT is an open‑source Python library that provides a multi‑agent framework for orchestrating LLMs as product managers, architects, engi…	45	70839	active
Open Interpreter 01 Open Interpreter is a terminal‑based coding agent that lets you interact with large language models to write, run, and debug code. It provi…	96	68541	active
Open Interpreter 01 Open Interpreter is a Rust-based terminal coding agent optimized for low-cost open models like Kimi K3, DeepSeek, and Qwen. It emulates pro…	96	68541	active
xtekky/gpt4free GPT4Free (g4f) is a Python-based application that aggregates multiple free LLM providers and offers a unified interface. It includes a web …	93	66757	active
AnythingLLM AnythingLLM is an all‑in‑one self‑hosted AI application that lets you chat with your documents, run AI agents, and use local LLMs. It provi…	96	66696	active
virattt/ai-hedge-fund An educational AI-powered hedge fund simulator where LLM-driven agents modeled after famous investors analyze stocks and make simulated tra…	87	63827	active
hugohe3/ppt-master PPT Master is an open‑source Python CLI that uses large language models to turn documents, web pages or topics into fully editable PowerPoi…	84	62770	active
mvanhorn/last30days-skill last30days is an AI agent skill that aggregates content from Reddit, X, YouTube, Hacker News, Polymarket and other web sources, then synthe…	80	62570	active
Pathway Live Data Framework Pathway is a Python ETL framework designed for stream processing, real-time analytics, and building LLM pipelines. It provides a high‑level…	94	62243	active
Alishahryar1/free-claude-code Free Claude Code is an open‑source Python CLI and library that lets you access multiple LLM coding agents (Claude, Codex, Pi, OpenCode, etc…	79	61286	active
MadsLorentzen/ai-job-search An AI-powered job application framework built on Claude Code that runs locally: it evaluates job postings, tailors CVs, writes cover letter…	83	60836	active
CrewAI CrewAI is an open‑source Python framework for building and orchestrating collaborative AI agents. It provides high‑level abstractions for c…	90	59546	active
karpathy/nanochat nanochat is a minimal, hackable full-stack LLM training harness that covers tokenization, pretraining, finetuning, evaluation, and inferenc…	59	58750	active
FoundationAgents/OpenManus OpenManus is an open-source framework for building general-purpose AI agents, created by the MetaGPT team as an open alternative to Manus. …	66	58639	active
jamiepine/voicebox Voicebox is an open‑source, local‑first AI voice studio that lets you clone voices, generate speech in many languages, and dictate into any…	74	57872	active
goose goose is an open-source, extensible general-purpose AI agent built in Rust that runs locally as a desktop app, CLI, and API. It connects to…	86	55171	active
microsoft/VibeVoice VibeVoice is an open‑source framework for high‑quality text‑to‑speech and speech‑to‑text generation. It combines autoregressive and diffusi…	60	54432	active
ggml-org/whisper.cpp A high-performance C/C++ port of OpenAI's Whisper automatic speech recognition model with no external dependencies. It supports CPU, GPU (M…	99	54397	stable
Cherry Studio Cherry Studio is a cross‑platform desktop client that provides a unified AI workstation with smart chat, autonomous agents, and hundreds of…	88	52721	active
moeru-ai/airi Project AIRI is an open-source, self-hosted AI companion (waifu/virtual character) platform inspired by Neuro-sama, supporting Live2D and V…	87	50294	active
Aider Aider is a command‑line tool that lets you pair program with large language models directly in your terminal. It integrates with git, suppo…	66	49387	active
Aider Aider is an open-source CLI tool that lets developers pair program with LLMs (Claude, GPT-4o, DeepSeek, Gemini, local models) directly in t…	66	49387	active
HKUDS/nanobot nanobot is an ultra‑lightweight, open‑source AI agent framework written in Python. It provides a WebUI, terminal, and chat‑app interfaces w…	82	49064	active
New API New API is a self-hosted LLM gateway and AI asset management system that aggregates 30+ model providers behind a single unified endpoint. I…	88	48905	active
exo-explore/exo exo is a Python application that turns a collection of devices (Macs, GPUs, etc.) into a distributed AI cluster for running large language …	82	47847	active
zhayujie/CowAgent CowAgent is an open‑source AI assistant and agent harness that can plan tasks, run tools and custom skills, and maintain a personal knowled…	98	47220	active
LibreChat LibreChat is an open‑source, self‑hosted ChatGPT‑style web application written in TypeScript. It provides a unified UI for multiple LLM pro…	85	46021	active
janhq/jan Jan is an open‑source desktop application that provides a ChatGPT‑like experience running large language models locally. It includes a GUI,…	96	44984	active
Wei-Shaw/sub2api Sub2API is a self-hosted AI API gateway written in Go that converts Claude, OpenAI, Gemini, and Grok subscriptions into unified API access.…	80	44264	active
LifeOS Fabric is an open-source framework for augmenting humans with AI by organizing crowdsourced AI prompts (called patterns) into a modular, re…	89	44214	active
chatanywhere/GPT_API_free GPT_API_free is a hosted API service that provides free access to multiple large language models such as GPT, DeepSeek, Claude, Gemini, and…	72	43890	active
LangGraph LangGraph is a low-level orchestration framework and runtime for building, managing, and deploying long-running, stateful AI agents in Pyth…	92	43532	stable
JustVugg/colibri Colibrì is a pure C inference engine that runs frontier Mixture-of-Experts language models by streaming weights from disk. It provides CLI …	81	42762	active
BettaFish MiroFish is a Python‑based multi‑agent AI engine that creates a high‑fidelity digital twin of real‑world scenarios. It lets users inject se…	75	42341	active
chatboxai/chatbox Chatbox is a cross‑platform desktop and mobile client for interacting with LLMs such as ChatGPT, Claude, Gemini and others. It provides a G…	96	41982	active
AstrBot AstrBot is an open-source AI agent assistant and development framework written in Python. It provides multi‑platform LLM chatbots with plug…	95	41689	active
AstrBot AstrBot is a multi-platform LLM chatbot and agentic AI assistant development framework written in Python. It integrates many instant messag…	95	41689	active
HKUDS/DeepTutor DeepTutor is an open‑source, agentic framework that combines large language models with retrieval‑augmented generation to deliver lifelong,…	79	41457	active
Hmbown/CodeWhale Codewhale is an open‑source terminal‑native coding agent written in Rust that can read, edit, and run code using any hosted or local LLM yo…	79	41097	active
datalab-to/marker Marker is an open‑source Python tool that converts PDFs and other document formats into markdown, JSON, HTML, and structured chunks. It sup…	89	40358	active
BitNet BitNet is a C++ inference framework for 1-bit large language models, providing both CPU and GPU kernels. It enables ultra‑low‑memory, high‑…	62	40295	active
BitNet bitnet.cpp is Microsoft's official inference framework for 1-bit large language models (BitNet b1.58), providing optimized CPU kernels with…	62	40295	active
THU-MAIC/OpenMAIC OpenMAIC is an open-source web application that provides an immersive multi‑agent learning environment. It lets users launch and interact w…	81	39947	active
lyogavin/airllm AirLLM is a Python library that enables running extremely large language models on a single low‑VRAM GPU. It achieves this without quantiza…	95	39932	active
mindsdb/mindshub MindsHub is an open‑source agent workspace that lets users run AI agents for research, analysis, content creation, and software development…	84	39759	active
QuivrHQ/quivr Quivr is an opinionated RAG (Retrieval-Augmented Generation) library, available as the quivr-core Python package, that lets developers inge…	76	39579	active
shiyu-coder/Kronos Kronos is an open‑source decoder‑only foundation model trained on financial candlestick (K‑line) data from dozens of global exchanges. It p…	50	39500	active
RSSNext/Folo Folo is an open-source AI-powered RSS reader that aggregates and summarizes content feeds across desktop and mobile platforms. It integrate…	85	39029	active
google/langextract LangExtract is a Python library that uses large language models to extract structured information from unstructured text. It provides preci…	85	38953	active
anthropics/financial-services A collection of reference AI agents, skills, and data connectors for financial-services workflows like investment banking, equity research,…	59	38868	active
linshenkx/prompt-optimizer Prompt Optimizer is an AI prompt optimization tool that helps users write, refine, test, and evaluate prompts for better LLM outputs. It sh…	89	38597	active
VectifyAI/PageIndex PageIndex is a Python SDK that provides a vectorless, reasoning‑based retrieval engine for long documents. It builds a hierarchical tree in…	86	38442	active
stanfordnlp/dspy DSPy is a Python library that lets you program language models using declarative signatures instead of hand‑written prompts. It provides mo…	93	38406	active
Claude Code Router Claude Code Router is a local control plane that routes requests from AI agents to multiple LLM providers and can fuse capabilities like vi…	85	37369	active
Claude Code Router Claude Code Router (CCR) is a self-hosted local model gateway and control plane that routes AI coding agent requests (Claude Code, Codex, G…	85	37369	active
ItzCrazyKns/Vane Vane is a self-hosted, privacy-focused AI answering engine that combines web search (via SearxNG) with LLMs to deliver cited answers. It su…	79	37362	active
songquanpeng/one-api One API is a self-hosted LLM API gateway and key management system that unifies OpenAI, Azure, Anthropic Claude, Google Gemini, DeepSeek, a…	52	37040	stable
reworkd/AgentGPT AgentGPT is a web application for assembling, configuring, and deploying autonomous AI agents directly in the browser. Users name a custom …	10	36296	active
DayuanJiang/next-ai-draw-io A Next.js web application that combines AI chat with draw.io diagram editing, letting users create and modify diagrams through natural lang…	87	36162	active
esengine/DeepSeek-Reasonix Reasonix is a DeepSeek-native AI coding agent that runs locally as a single Go binary. It provides a terminal‑first workflow with optional …	78	35837	active
HKUDS/Vibe-Trading Vibe-Trading is an open-source AI finance research agent that turns natural-language prompts into runnable market research, backtests, and …	82	35591	active
virgiliojr94/book-to-skill book-to-skill is a Python CLI that transforms technical books and documents into Claude Code agent skills. It extracts content, converts it…	78	34574	active
Tabby Tabby is a self‑hosted AI coding assistant that provides LLM‑powered code completion, inline chat, and answer engine capabilities. It runs …	76	33888	active
Tabby Tabby is a self-hosted, open-source AI coding assistant that serves as an on-premises alternative to GitHub Copilot. It provides LLM-powere…	76	33888	active
SillyTavern SillyTavern is a locally installed, web‑based user interface for interacting with text‑generation LLMs, image generators, and TTS models. I…	91	33851	active
SillyTavern SillyTavern is a locally installed, self-hosted web UI for interacting with text generation LLMs, image generation engines, and TTS voice m…	91	33851	active
alchaincyf/nuwa-skill Nuwa.skill is an Agent Skills-compatible package that distills how a person thinks—mental models, decision heuristics, and expression style…	56	33624	active
cursor/cursor Cursor is an AI‑powered coding assistant that can understand, plan, write, and debug code across multiple languages. It provides a desktop …	64	33264	active
tashfeenahmed/freellmapi FreeLLMAPI is an open‑source, self‑hosted router that aggregates free tiers from dozens of LLM providers behind a single OpenAI‑compatible …	82	32961	active
onyx-dot-app/onyx Onyx is an open‑source AI platform that provides a chat interface powered by large language models and a context layer that indexes knowled…	93	32566	active
hsliuping/TradingAgents-CN A Chinese-enhanced fork of the TradingAgents multi-agent LLM framework for stock analysis and trading research, supporting A-shares, Hong K…	62	32318	active
p-e-w/heretic Heretic is a command-line tool that automatically removes censorship (safety alignment) from transformer-based language models using direct…	73	32267	active