Open-source llm-inference software

3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.

3358 products, primary matches first, then adoption-weighted; health v2 shown.

Product	Health v2	Stars	Maturity
tekaratzas/RustGPT A transformer-based large language model implemented entirely in pure Rust with no external ML frameworks, using only ndarray for matrix op…	35	3175	active
nesaorg/nesa Nesa is a privacy-preserving AI inference platform that runs models like Llama, Mistral, and Stable Diffusion with end-to-end encryption us…	22	3163	active
MaxMiksa/Auto-Company Auto Company is a fully autonomous AI company simulation that runs 24/7 on your own PC, orchestrating 14 autonomous AI agents modeled after…	83	3156	active
FB208/OpenBidKit_Yibiao OpenBidKit Yibiao (易标) is an open-source Electron desktop application that uses LLMs and RAG to generate bid/tender proposal documents from…	81	3149	active
BayLing-Models/BayLing-Speech LLaMA-Omni is an end-to-end speech interaction model built on Llama-3.1-8B-Instruct that generates simultaneous text and speech responses f…	30	3144	active
hero8152/Infinite-Canvas An infinite canvas desktop application for orchestrating AI image, video, and LLM generation workflows. It integrates with ComfyUI, OpenAI-…	55	3143	active
CyberStrikeus/CyberStrike CyberStrike is an open-source AI-powered offensive security harness that runs automated penetration testing from the terminal. It orchestra…	79	3141	active
HeyWillow/willow Willow is an open source, self-hosted voice assistant platform for ESP32-S3-BOX hardware, designed as a privacy-focused alternative to Amaz…	84	3127	active
MiroFlow MiroFlow is a performance-first open-source agent framework for building, orchestrating, and benchmarking LLM-powered agents, with MiroThin…	56	3120	active
cocktailpeanut/dalai Dalai is a self-hosted application that makes it simple to run LLaMA (and Alpaca) language models locally with a one-command installer and …	22	12887	maintenance
SciSharp/BotSharp BotSharp is an open-source multi-agent AI framework written in C# for .NET, providing an agent abstraction layer, conversation state manage…	77	3114	active
rpamis/comet Comet is an npm-distributed CLI and Skill platform that gives AI coding agents a resumable, phase-guarded workflow for long-running tasks, …	79	3113	active
b4rtaz/distributed-llama A C++ application that connects multiple home devices into a cluster for distributed LLM inference using tensor parallelism. It supports Ll…	73	3100	active
AutoCodeRoverSG/auto-code-rover AutoCodeRover is an autonomous software engineering agent that combines LLMs with code structure analysis to automatically resolve GitHub i…	20	3100	active
TensorFlow Lite for Microcontrollers TensorFlow Lite for Microcontrollers (TFLM) is a C++ port of TensorFlow Lite for running ML models on microcontrollers, DSPs, and other mem…	76	3098	active
Litho (deepwiki-rs) Litho (deepwiki-rs) is a Rust-based AI-powered documentation generator that analyzes codebases and produces C4-model architecture documenta…	81	3096	active
DeepSeek PHP Chat A lightweight PHP demo application that calls DeepSeek/OpenAI-compatible chat APIs with streaming (EventSource) output, supporting Markdown…	33	3094	active
ridgerchu/matmulfreellm A Python implementation of MatMul-Free LM, a language model architecture that eliminates matrix multiplication operations using ternary wei…	76	3093	active
robert-mcdermott/ai-knowledge-graph A Python CLI tool that uses an LLM to extract Subject-Predicate-Object triplets from unstructured text documents and renders them as an int…	84	3088	active
Project-N-E-K-O/N.E.K.O Project N.E.K.O. is an open-source AI companion application — a proactive catgirl-style AI that lives on your desktop, initiates interactio…	84	3087	active
containers/ramalama RamaLama is a Python CLI tool that simplifies serving and running AI/LLM models locally by leveraging OCI containers (Podman/Docker). It au…	85	3079	active
sonos/tract Tract is Sonos' tiny, self-contained neural-network inference engine written in Rust. It loads ONNX, TensorFlow/TFLite, and NNEF models, op…	97	3075	active
kairos-agi/kairos Kairos is the official open-source implementation of a 4B-parameter native cross-embodiment world model that unifies video understanding, f…	59	3066	active
MiniMax-AI/Mini-Agent A minimal yet professional Python demo project showcasing best practices for building a single LLM agent on the MiniMax M2.5 model via an A…	44	3060	active
swirlai/swirl-search SWIRL is an open-source federated AI search and RAG platform that queries enterprise apps and data sources live, re-ranks results with a hy…	92	3046	active
Natively-AI-assistant/natively-cluely-ai-assistant Natively is a free, source-available desktop AI meeting assistant and interview copilot that provides real-time transcription, AI-generated…	83	3036	active
ianarawjo/ChainForge ChainForge is an open-source visual programming environment for battle-testing prompts to LLMs, built on ReactFlow and Flask. It lets users…	92	3035	active
allweonedev/presentation-ai An open-source, AI-powered presentation generator (a Gamma.app alternative) built with Next.js that creates professional, fully editable sl…	55	3033	active
x-dr/chatgptProxyAPI A self-deployable reverse proxy for the OpenAI API built on Cloudflare Workers and Cloudflare Pages, letting users replace api.openai.com w…	10	3029	active
dw-dengwei/daily-arXiv-ai-enhanced A tool that automatically crawls daily arXiv papers and summarizes them using LLMs like DeepSeek, publishing the results as a browsable sit…	62	3027	active
luminal-ai/luminal Luminal is a high-performance general-purpose ML inference compiler written in Rust that lowers models to a minimal 15-op dataflow IR and c…	77	3026	active
betalgo/openai A community .NET library (Betalgo.Ranul.OpenAI) for accessing OpenAI's API, including ChatGPT, DALL-E, Whisper, and Azure OpenAI. It lets .…	63	3021	active
TheBlewish/Automated-AI-Web-Researcher-Ollama A Python application that turns a locally-run Ollama LLM into an automated web research assistant. Given a single query, it generates focus…	64	3017	active
pytorch/TensorRT Torch-TensorRT is a compiler library that accelerates PyTorch model inference on NVIDIA GPUs using TensorRT. It supports just-in-time compi…	93	3009	active
thu-pacman/chitu Chitu (赤兔) is a high-performance inference framework for large language models, positioned as a production-grade LLM serving engine. It sup…	82	3009	active
leigest519/OpenGame OpenGame is an open-source agentic framework for end-to-end web game creation from a single prompt, built around a reusable 'Game Skill' sy…	56	3009	active
leigest519/ScreenCoder ScreenCoder is a UI-to-code generation system that converts screenshots or design mockups into clean, editable HTML/CSS using a modular mul…	61	3006	active
eli64s/readme-ai ReadmeAI is a Python CLI tool that automatically generates README.md files for any codebase using large language models. It supports multip…	66	3004	active
J-Space Cognition Suite A model-agnostic inference-time control suite packaged as an installable AI agent Skill (SKILL.md with modules, references, and Python veri…	77	3002	active
pytorch/ao TorchAO is a PyTorch-native library for model optimization through quantization and sparsity. It supports quantizing weights, gradients, op…	87	3001	active
Dicklesworthstone/llm_aided_ocr A Python tool that converts scanned PDFs to text via Tesseract OCR, then uses LLMs (local or API-based like OpenAI/Anthropic) to correct OC…	68	3001	active
signerless/llm-checker A Node.js CLI tool that scans your hardware (GPU, VRAM, CPU, memory) and recommends which LLM or small language models you can run locally,…	80	2998	active
oritera/Cairn Cairn is a general-purpose AI state-space search engine built on a blackboard architecture with a fact-intent graph, where LLM agent worker…	71	2997	active
vLLM vllm-ascend is a community-maintained hardware plugin that enables vLLM to run large language model inference on Huawei Ascend NPUs. It imp…	83	2996	active
ulab-uiuc/LLMRouter LLMRouter is an open-source Python library that intelligently routes queries to the most suitable LLM based on task complexity, cost, and p…	62	2995	active
Flow2API Flow2API is a self-hosted Python/FastAPI service that exposes an OpenAI- and Gemini-compatible API on top of Google Flow (VideoFX/ImageFX) …	60	2989	active
Biohub/esm A Python library and model release providing ESMC protein language models, ESMFold2 structure prediction, and the ESM Atlas of 6.8 billion …	88	2987	active
OpenMOSS/MOSS MOSS is an open-source tool-augmented conversational large language model from Fudan University, released with base models, SFT models, plu…	72	12253	maintenance
omnara-ai/omnara Omnara is an open-source platform for running managed, durable AI agents, handling execution and state while you bring your own models, too…	82	2984	active
togethercomputer/MoA Together AI's Mixture-of-Agents (MoA) is a Python implementation of a layered LLM architecture where multiple open-source models answer pro…	25	2982	active
smol-ai/developer smol developer is a 'junior developer' AI agent that scaffolds an entire codebase from a product spec, and can be embedded as a library to …	30	12188	maintenance
michaelfeil/infinity Infinity is a high-throughput, low-latency serving engine that exposes text-embedding, reranking, CLIP, CLAP and ColPali models via an Open…	96	2966	active
huggingface/tau Tau is a small, readable terminal coding agent written in Python, inspired by Pi, that can read files, edit code, run commands, and maintai…	81	2966	active
ax-llm/ax Ax is a TypeScript-first LLM programming framework implementing the DSPy model, providing typed signatures, agents, flows, and optimizers l…	93	2961	active
li-plus/chatglm.cpp A pure C++ implementation of ChatGLM-6B, ChatGLM2-6B, ChatGLM3, and GLM-4(V) based on ggml, enabling memory-efficient real-time chatting on…	21	2957	active
database.build An in-browser Postgres sandbox where each database runs locally via PGlite (WASM Postgres) and is paired with an LLM assistant for generati…	58	2953	active
MacPaw/OpenAI A community-maintained Swift package that wraps the OpenAI public API, supporting chat completions, responses, function calling, MCP tools,…	100	2952	active
turboderp/exllama ExLlama is a standalone Python/C++/CUDA implementation of the Llama architecture optimized for running 4-bit GPTQ quantized weights on mode…	30	2947	active
Luce-Org/lucebox Lucebox is an open-source LLM inference engine written in C++ with custom CUDA/HIP kernels, focused on speculative decoding and speculative…	60	2941	active
memodb-io/memobase Memobase is a user profile-based long-term memory system for LLM applications, providing per-user profiles and event timelines that chatbot…	47	2936	active
OpenMind/OM1 OM1 is a modular AI runtime and hardware abstraction layer for building multimodal AI agents that run on physical robots and in simulators.…	77	2933	active
GodsScion/Auto_job_applier_linkedIn A Python/Selenium desktop application that automates LinkedIn Easy Apply job applications, finding relevant jobs, filling application quest…	93	2931	active
zjunlp/EasyEdit EasyEdit is an easy-to-use knowledge editing framework for large language models, supporting methods that modify model parameters or steer …	74	2929	active
ErlichLiu/DeepClaude DeepClaude is a self-hostable service that chains DeepSeek R1's reasoning output with Claude Sonnet or Gemini models to produce a combined …	48	2928	active
InternLM/InternLM-XComposer InternLM-XComposer is a family of large vision-language models from the InternLM team, including XComposer2.5 for long-context multimodal u…	35	2927	active
langchain-ai/social-media-agent A TypeScript AI agent built on LangGraph that takes a URL, scrapes its content, and generates Twitter and LinkedIn posts using LLMs. It inc…	67	2920	active
JunChenMoCode/ChatGPT_JCM A Vue2 + ElementUI web management interface that aggregates OpenAI API endpoints (models, chat, images, audio, fine-tuning, files) into a g…	22	2917	active
ModelTC/LightX2V LightX2V is a lightweight, high-performance inference framework for image and video generation, supporting tasks like text-to-video, image-…	83	2914	active
xiao-zaiyi/illa-helper A browser extension that helps users learn languages naturally while browsing the web, based on the comprehensible input theory. It provide…	73	2899	active
MartialBE/one-hub One Hub is a self-hosted OpenAI-compatible API management and distribution system forked from songquanpeng/one-api, written in Go. It aggre…	62	2896	active
ideogram-oss/ideogram4 Ideogram 4 is an open-weight text-to-image foundation model trained from scratch, with inference code and weights released in Python. It fe…	51	2888	active
GCWing/BitFun BitFun is a cross-platform desktop AI agent application with a high-performance Rust agent runtime that writes code, produces documents, an…	84	2876	active
Y-Research-SBU/QuantHarness QuantHarness is a multi-agent LLM framework for high-frequency trading analysis, built on LangChain and LangGraph. Specialized agents compu…	59	2874	active
Amurex Amurex is an open-source Chrome extension that acts as an AI meeting copilot for Google Meet and MS Teams. It provides real-time suggestion…	31	2873	active
moltis-org/moltis Moltis is a secure, self-hosted personal AI agent server written in Rust that ships as a single binary with sandboxed execution, multi-prov…	75	2867	active
Prism-Shadow/penguin-harness PenguinHarness is an open-source, local-first multi-agent application development platform that automates building, optimizing, and deployi…	81	2864	active
VRSEN/OpenSwarm OpenSwarm is an open-source multi-agent system built on Agency Swarm that turns a single terminal prompt into complete deliverables like sl…	71	2861	active
anysphere/priompt Priompt is a JSX-based prompt design library for building LLM prompts with React-like components. It uses priority levels to decide which p…	47	2856	active
songguoxs/seedance-prompt-skill A Claude Code custom skill that turns Claude into an AI video prompt engineer for ByteDance's Seedance 2.0 video generation platform. It co…	42	2849	active
rednote-machine-learning/RedKnot RedKnot is a long-context LLM inference acceleration library built on SGLang, using head-classified KV reuse, offline KV storage with RoPE …	57	2840	active
spcl/graph-of-thoughts Official Python implementation of the Graph of Thoughts (GoT) framework from ETH Zurich, which models complex problem-solving as a Graph of…	48	2840	active
antirez/h3.c A native C inference engine for the MiniMax H3 model on Apple Silicon, using Metal for GPU acceleration. It generates video (and audio) fro…	53	2828	active
Doriandarko/o1-engineer o1-engineer is a Python command-line tool that uses OpenAI's API to help developers generate code, edit files, plan projects, and review co…	23	2826	active
trustgraph-ai/trustgraph TrustGraph is an open-source context interoperability layer that builds unified semantic context layers using hypergraphs, knowledge graphs…	78	2808	active
facebookresearch/HyperAgents A research framework from Meta AI implementing self-referential, self-improving LLM agents that can optimize their own code for arbitrary c…	54	2803	active
fuergaosi233/claude-code-proxy A Python proxy server that translates Claude API requests (/v1/messages) into OpenAI-compatible API calls, letting the Claude Code CLI work…	47	2799	active
ryfineZ/codex-session-patcher A Python tool that cleans AI refusal responses from Codex CLI, Claude Code, and OpenCode session files so conversations can be resumed, wit…	73	2797	active
OpenDCAI/Paper2Any Paper2Any is an open-source Python application that turns research papers, text, or topics into editable research figures, technical route …	59	2797	active
Mega4alik/ollm oLLM is a lightweight Python library for large-context LLM inference built on Hugging Face Transformers and PyTorch. It offloads weights an…	57	2795	active
zachlatta/freeflow FreeFlow is a free, open-source macOS dictation app that transcribes speech via Groq's fast API and pastes cleaned text into any focused te…	74	2794	active
openlit/openlit OpenLIT is an open-source AI engineering platform providing OpenTelemetry-native LLM observability, evaluations, guardrails, prompt managem…	86	2792	active
Vali-98/ChatterUI ChatterUI is a native mobile frontend for running large language models on-device via llama.cpp or connecting to commercial and open-source…	89	2790	active
vava-nessa/free-coding-models A CLI tool that tracks a large catalog of free coding LLM models across 20+ providers in real time, benchmarking their speed and stability.…	77	2790	active
HarleyCoops/Math-To-Manim-lfs-archive A Python project that turns math and physics questions into reasoned visual explanations rendered as Manim animations and study notes. It u…	80	2789	active
folke/sidekick.nvim A Neovim plugin that integrates Copilot LSP's Next Edit Suggestions with an embedded terminal for AI CLI tools like Claude Code, Gemini CLI…	71	2788	active
filip-michalsky/SalesGPT SalesGPT is an open-source, context-aware AI sales agent built with LLMs that automates sales outreach across voice, email, SMS, and messag…	21	2778	active
Promptflow Prompt flow is a suite of Python SDK, CLI, and VS Code extension tools for the end-to-end development of LLM-based AI applications, linking…	76	11254	maintenance
FasterDecoding/Medusa Medusa is a framework that accelerates LLM text generation by adding multiple decoding heads to an existing model, avoiding the need for a …	19	2776	active
icereed/paperless-gpt A self-hosted companion application for paperless-ngx that uses LLMs and vision models to auto-generate document titles, tags, and dates, a…	90	2773	active
google-research/tabfm TabFM is a scikit-learn compatible tabular foundation model from Google Research that performs zero-shot classification and regression on t…	63	2769	active