Open-source llm-inference software
3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
3358 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| kijai/ComfyUI-CogVideoXWrapper A ComfyUI custom node wrapper for CogVideoX and related video generation models (including Fun variants, CogVideoX 1.5, and Go-with-the-Flo… | 36 | 1546 | active |
| baichuan-inc/Baichuan-7B Baichuan-7B is an open-source, commercially usable 7-billion-parameter pretrained language model built on the Transformer architecture, tra… | 30 | 5648 | maintenance |
| PaperDebugger/paperdebugger PaperDebugger is an AI-powered academic writing assistant delivered as a Chrome extension that integrates directly into Overleaf for in-edi… | 57 | 1542 | active |
| KruxAI/ragbuilder RagBuilder is a Python toolkit that automatically builds an optimal, production-ready Retrieval-Augmented Generation (RAG) pipeline for you… | 65 | 1539 | active |
| ATH-MaaS/Marco-o1 Marco-o1 is an open large reasoning model from Alibaba International Digital Commerce, designed for o1-like chain-of-thought reasoning acro… | 58 | 1537 | active |
| ggml-org/llama.vscode A VS Code extension providing local LLM-assisted text and code completion, AI chat, and agentic coding powered by llama.cpp. It supports au… | 84 | 1536 | active |
| tldraw/make-real-starter A starter template for building 'Make Real' style apps that turn hand-drawn UI mockups on a tldraw infinite canvas into working HTML using … | 60 | 1536 | active |
| Mirascope Mirascope is a Python (and TypeScript) library providing a unified, provider-agnostic interface for calling frontier LLMs like OpenAI, Anth… | 91 | 1535 | active |
| Nativ Nativ is a free, MIT-licensed macOS desktop application for running open AI models locally on Apple Silicon Macs, built on MLX-VLM. It prov… | 81 | 1534 | active |
| razzant/ouroboros Ouroboros is an open-source, general-purpose AI agent with persistent identity, durable memory, and the ability to rewrite its own implemen… | 78 | 1533 | active |
| bugbasesecurity/pentest-copilot Pentest Copilot is an open-source, AI-driven penetration testing agent that connects to a Kali attack box, autonomously runs security tools… | 61 | 1531 | active |
| techjarves/OpenClaude-Portable A portable AI coding agent that runs Claude Code-style functionality from a USB drive or any folder with no installation, bundling a self-c… | 57 | 1531 | active |
| ibelick/zola Zola is an open-source, self-hostable chat interface for interacting with multiple AI models including OpenAI, Claude, Gemini, Mistral, and… | 42 | 1531 | active |
| ATH-MaaS/Ovis Ovis is an open-source Multimodal Large Language Model (MLLM) architecture that structurally aligns visual and textual embeddings, with rel… | 62 | 1528 | active |
| David-Kunz/gen.nvim A Neovim plugin that generates text using LLMs (primarily via Ollama) with customizable prompts. It streams model responses into editor win… | 33 | 1527 | active |
| kyutai-labs/unmute Unmute is a system that lets any text LLM listen and speak by wrapping it with Kyutai's low-latency speech-to-text and text-to-speech model… | 61 | 1525 | active |
| beehive-lab/TornadoVM TornadoVM is a Java programming framework that JIT-compiles JVM bytecode into GPU kernels targeting NVIDIA CUDA, OpenCL, and Apple Metal at… | 98 | 1523 | active |
| Coframe/coffee Coffee is an AI-powered tool for building and iterating on React UI components directly from your IDE. It uses LLMs to generate and refine … | 64 | 1523 | active |
| stepfun-ai/Step-Audio2 Step-Audio 2 is an end-to-end multimodal large language model for industry-strength audio understanding and speech conversation, with open-… | 48 | 1521 | active |
| mattzh72/articraft Articraft is a research tool that generates articulated 3D assets from text prompts or reference images by having an LLM write Python using… | 62 | 1519 | active |
| PrathamLearnsToCode/paper2code An agent skill for coding agents like Claude Code that converts an arXiv paper URL into a working Python implementation. It generates citat… | 45 | 1519 | active |
| six2dez/burp-ai-agent Custom AI Agent (formerly Burp AI Agent) is a Burp Suite extension written in Kotlin that integrates LLMs into web security workflows via l… | 77 | 1517 | stable |
| kodustech/kodus-ai Kodus (Kody) is an open-source AI code review agent that reviews pull requests on GitHub, GitLab, Bitbucket, and Azure Repos. It is model-a… | 84 | 1514 | active |
| andrewnguonly/Lumos Lumos is a Chrome extension that acts as a RAG-powered LLM co-pilot for browsing the web, letting users summarize pages and ask questions a… | 18 | 1511 | active |
| microsoft/ai-dev-gallery AI Dev Gallery is a Windows application from Microsoft that lets developers explore over 25 interactive samples powered by local AI models … | 67 | 1510 | active |
| NPC-Worldwide/npcpy npcpy is a Python library providing primitives for building applications with multimodal LLMs, agentic AI, and knowledge graphs. It support… | 84 | 1509 | active |
| ovg-project/kvcached kvcached is a Python library that brings OS-style virtual memory abstraction to KV cache management for LLM serving and training on shared … | 72 | 1509 | active |
| jgravelle/AutoGroq AutoGroq is a Python application that dynamically generates tailored teams of AI agents for Autogen and similar frameworks based on user pr… | 16 | 1508 | active |
| imoneoi/openchat OpenChat is a library of open-source large language models fine-tuned with C-RLFT, an offline reinforcement learning strategy that learns f… | 20 | 5485 | maintenance |
| TIGER-AI-Lab/TheoremExplainAgent An agentic AI system that generates long-form (5+ minute) Manim animation videos explaining mathematical and STEM theorems using LLM agents… | 32 | 1507 | active |
| appleboy/CodeGPT CodeGPT is a Go-based CLI tool that uses LLM APIs (OpenAI, Gemini, Anthropic, Groq, Ollama, OpenRouter) to generate git commit messages and… | 96 | 1506 | active |
| langchain-ai/open-canvas Open Canvas is an open-source web application for collaborating with LLM agents to write documents and code, inspired by OpenAI's Canvas. I… | 10 | 5477 | maintenance |
| uzairansaruzi/hermex Hermex is a native SwiftUI iPhone app for controlling a self-hosted Hermes (hermes-webui) AI agent from your phone. It provides streaming c… | 81 | 1505 | active |
| openonion/connectonion ConnectOnion is a Python framework for building production-ready AI agents with tools, plugins, event hooks, and multi-agent collaboration.… | 82 | 1499 | active |
| joewongjc/type4me Type4Me is a macOS voice input method that transcribes speech in real time (streaming, local or cloud ASR engines) and refines text with LL… | 83 | 1498 | active |
| fynnfluegge/rocketnotes Rocketnotes is a web-based Markdown note-taking app with integrated AI features such as chat with your documents, text completion, voice-to… | 66 | 1495 | active |
| fawney19/Aether Aether is a self-hosted AI API gateway written in Rust that provides a unified entry point for Claude, OpenAI, Gemini, and their CLI client… | 79 | 1492 | active |
| octotools/octotools OctoTools is an open-source, training-free agentic framework from Stanford that augments large language models with extensible tools for co… | 53 | 1490 | active |
| yusufcanb/tlm tlm is a local CLI copilot powered by Ollama that suggests shell commands, explains them, and answers questions with context using open-sou… | 55 | 1488 | active |
| dust-tt/dust Dust is an open-source platform for building, deploying, and managing custom AI agents powered by large language models. It combines a web … | 95 | 1484 | active |
| OpenGenerativeAI/llm-colosseum LLM Colosseum is a benchmarking tool that pits large language models against each other in real-time Street Fighter III matches to evaluate… | 18 | 1482 | active |
| LLM Vision LLM Vision is a Home Assistant integration (installed via HACS) that uses multimodal large language models to analyze images, videos, live … | 88 | 1479 | active |
| Vizards/deepseek-v4-for-copilot A VS Code extension that adds DeepSeek V4 Flash, Pro, and Flash Vision Exp models to the GitHub Copilot Chat model picker. It lets develope… | 79 | 1479 | active |
| morsoli/aimangastudio AIMangaStudio is a web application that uses AI (Google GenAI) to create manga/comics end-to-end, including script generation, storyboard l… | 58 | 1479 | active |
| theroyallab/tabbyAPI TabbyAPI is a FastAPI-based OpenAI-compatible API server for running large language models locally using the ExllamaV3 backend on NVIDIA GP… | 72 | 1478 | active |
| salesforce/CodeTF CodeTF is a Python transformer library for code large language models, providing unified interfaces for training, fine-tuning, and inferenc… | 10 | 1478 | active |
| qwersyk/Newelle Newelle is a GTK4/GNOME virtual assistant application for Linux that connects to local (Ollama, Llama.cpp) and cloud LLM providers. It supp… | 92 | 1477 | active |
| Nutlope/napkins Napkins.dev is an open-source web application that turns screenshots or wireframes of website designs into working React + Tailwind code us… | 60 | 1477 | active |
| Om-Alve/smolGPT A minimal pure-PyTorch implementation for training small GPT-style LLMs from scratch, featuring flash attention, RMSNorm, SwiGLU, RoPE, and… | 21 | 1474 | active |
| NousResearch/Hermes-Function-Calling A repository providing function-calling capabilities for the Hermes Pro large language models, including stock price and financial data too… | 49 | 1473 | active |
| shenminglinyi/PlotPilot PlotPilot (墨枢) is an open-source narrative engine kernel for long-form AI-assisted fiction and script writing, built in Python with FastAPI… | 72 | 1472 | active |
| RUC-NLPIR/WebThinker WebThinker is a research framework that empowers large reasoning models (like DeepSeek-R1 and QwQ) with deep research capability, letting t… | 41 | 1470 | active |
| HexmosTech/git-lrc git-lrc is a free, open-source tool that runs micro AI code reviews automatically on every git commit. It is written in Go and integrates w… | 82 | 1469 | active |
| fictionco/fiction Fiction is a next-generation CMS and application framework for building AI-generated content, personal websites, marketing funnels, and new… | 40 | 1468 | active |
| JailbrokenAI/wallbreaker Wallbreaker is a Claude-Code-style terminal harness for red-teaming LLMs, driving an autonomous agent loop that runs jailbreak attacks (PAI… | 54 | 1464 | active |
| Danau5tin/multi-agent-coding-system A Python multi-agent AI coding system where an orchestrator agent dispatches explorer and coder subagents with shared knowledge artifacts, … | 28 | 1463 | active |
| JoasASantos/NeuroSploit NeuroSploit is an AI-powered penetration testing framework written in Rust that turns a URL, repository, app, or host into an autonomous se… | 86 | 1462 | active |
| gabotechs/MusicGPT MusicGPT is a Rust application that runs music generation AI models (currently Meta's MusicGen) locally, generating music from natural lang… | 27 | 1462 | active |
| hotovo/aider-desk AiderDesk is an open-source desktop application (Electron) that provides a GUI and orchestration layer for AI-assisted software development… | 85 | 1460 | active |
| langchain-ai/openwork Openwork is an Electron-based desktop interface for deepagentsjs, providing a harness for building and running deep AI agents with filesyst… | 70 | 1458 | active |
| webbrain-one/webbrain WebBrain is an open-source AI browser agent extension for Chrome, Firefox, and Edge that puts an LLM-powered assistant in a side panel. It … | 78 | 1456 | active |
| acon96/home-llm A Home Assistant custom integration plus fine-tuned small language models that let you control your smart home entirely with a local LLM, n… | 93 | 1455 | active |
| jekalmin/extended_openai_conversation A Home Assistant custom component that acts as a conversation agent powered by OpenAI. It uses OpenAI's function calling to control Home As… | 95 | 1454 | active |
| InternAgent InternAgent-1.5 is a Python-based unified agentic framework for long-horizon autonomous scientific discovery, orchestrating multi-agent sys… | 58 | 1454 | active |
| Paritok-official/paritok-4b-v1 Paritok is a non-destructive compression gateway that sits as a drop-in proxy between AI coding agents (Claude Code, Cursor, Codex, OpenHan… | 56 | 1454 | active |
| sauravpanda/BrowserAI BrowserAI is a TypeScript library for running LLMs, speech recognition, text-to-speech, and audio separation models directly in the browser… | 73 | 1452 | active |
| wordware-ai/twitter A Next.js web application that generates AI-powered personality profiles and compatibility analyses from Twitter/X handles using an LLM age… | 64 | 1450 | active |
| vitalops/openvibe Openvibe is a Python implementation of opencode, an open-source AI coding agent that runs terminal commands, edits files, and executes agen… | 58 | 1450 | active |
| AgentAlphaAGI/Idea2Paper Idea2Paper is an end-to-end multi-agent framework that decomposes the research process into structured phases, turning raw research ideas i… | 44 | 1450 | active |
| wannabespace/conar Conar (formerly Tamery) is an open-source, AI-powered desktop database client built with Electron for PostgreSQL, MySQL, MSSQL, and ClickHo… | 78 | 1449 | active |
| opencx-labs/OpenChat OpenChat is a self-hosted chatbot console that lets everyday users create and manage unlimited custom chatbots powered by GPT-3/GPT-4 with … | 10 | 5228 | maintenance |
| ChatGPT MD An Obsidian community plugin that integrates ChatGPT-style AI chat directly into notes, supporting wiki-link context, AI agents, and tool c… | 93 | 1447 | active |
| Nutlope/pdftochat PDFToChat is an open-source Next.js web application that lets users upload PDFs and chat with them using AI. It uses Together AI's Mixtral … | 65 | 1443 | active |
| google/oss-fuzz-gen A Google framework that uses large language models to automatically generate fuzz targets for real-world C/C++, Java, and Python projects, … | 55 | 1443 | active |
| deepseek-ai/EPLB EPLB is DeepSeek's open-source Expert Parallelism Load Balancer for Mixture-of-Experts models. It computes balanced expert replication and … | 23 | 1443 | active |
| perminder-klair/subwave SUB/WAVE is a self-hostable personal internet radio station that broadcasts a single shared Icecast stream to all listeners simultaneously.… | 78 | 1439 | active |
| edwko/OuteTTS OuteTTS is a Python interface for running OuteAI's text-to-speech models, supporting multiple inference backends including llama.cpp, Huggi… | 49 | 1437 | active |
| eumemic/ai-legion AI Legion is a TypeScript framework for running LLM-powered autonomous agents that collaborate to accomplish tasks. Agents are managed thro… | 37 | 1437 | active |
| salesforce/CodeGen CodeGen is a family of open-source large language models (350M to 16B parameters) from Salesforce AI Research for program synthesis, genera… | 67 | 5181 | maintenance |
| Uncensored AI Studio Uncensored-Local-Studio is a portable, fully offline AI assistant stack (Ollama + Llama 3 / other GGUF models + AnythingLLM) that runs enti… | 57 | 1436 | active |
| milanglacier/minuet-ai.nvim Minuet AI is a Neovim plugin that provides AI-powered code completion as you type, using LLM providers such as OpenAI, Claude, Gemini, Code… | 86 | 1435 | active |
| Blueturboguy07/cue cue is an open-source desktop AI copilot that floats a glass panel over your screen, capturing your screen, microphone, and meeting audio t… | 78 | 1431 | active |
| jakobhoeg/nextjs-ollama-llm-ui A fully-featured, ChatGPT-inspired web interface for chatting with local Ollama LLMs, built with Next.js, React, and Tailwind. It runs full… | 24 | 1426 | active |
| ammaarreshi/gemma-chat An open-source Electron app that runs Google's Gemma 4 models locally on Apple Silicon via Apple's MLX framework, providing a chat interfac… | 47 | 1425 | active |
| shobrook/wut wut is a Python CLI tool that sends the output of your last terminal command to an LLM (OpenAI, Claude, or a local Ollama model) and return… | 23 | 1425 | active |
| WPeGPT WPeGPT is an IDA Pro plugin that integrates LLM models (OpenAI, DeepSeek, or any OpenAI-compatible API) into binary analysis workflows. It … | 69 | 1423 | active |
| myshell-ai/AIlice AIlice is a fully autonomous, general-purpose AI agent built on open-source LLMs, designed as a JARVIS-like standalone assistant. It uses a… | 31 | 1418 | active |
| callstackincubator/ai A collection of on-device AI primitives for React Native offering Vercel AI SDK-compatible APIs for running LLMs, embeddings, transcription… | 70 | 1415 | active |
| SakanaAI/ShinkaEvolve ShinkaEvolve is a Python framework that combines LLMs with evolutionary algorithms to automatically evolve and improve scientific code. It … | 76 | 1414 | active |
| Terrain Diffusion Terrain Diffusion is a Python framework that uses diffusion models as a learned, deterministic replacement for Perlin noise, generating inf… | 58 | 1414 | active |
| SpikingBrain SpikingBrain-7B is a brain-inspired large language model that combines hybrid efficient attention, MoE modules, and spike encoding, with a … | 51 | 1408 | active |
| awaescher/OllamaSharp OllamaSharp is a .NET client library providing bindings for the entire Ollama API, including chat, generation, embeddings, and model manage… | 89 | 1407 | active |
| agentenv/agentflow AgentFlow is a Python library for orchestrating large numbers of LLM agents (Codex, Claude, Kimi, Pi) as programmatic dependency graphs wit… | 58 | 1407 | active |
| kizuna-ai-lab/sokuji Sokuji is a cross-platform real-time two-way speech translation app for bilingual meetings, available as a desktop application (Windows, ma… | 84 | 1406 | active |
| robertpiosik/CodeWebChat Code Web Chat (CWC) is a free, privacy-first VS Code extension toolkit for AI-assisted coding that builds unified prompts with static conte… | 67 | 1406 | active |
| XiaoMi/mace MACE (Mobile AI Compute Engine) is a deep learning inference framework optimized for mobile heterogeneous computing on Android, iOS, Linux … | 23 | 5052 | maintenance |
| jofizcd/Soul-of-Waifu Soul of Waifu is an open-source desktop application for creating AI companion characters with Live2D/VRM avatars, voice chat, and persisten… | 89 | 1404 | active |
| zubair-trabzada/ai-sales-team-claude A collection of 14 Claude Code skills and 5 parallel agents that turn Claude Code into an AI-powered sales team. It researches prospects, q… | 45 | 1404 | active |
| RahulSChand/gpu_poor A web-based calculator that estimates GPU memory requirements and inference/finetuning throughput (token/s) for any LLM. It supports quanti… | 19 | 1401 | active |
| 0nutation/SpeechGPT SpeechGPT is a series of speech large language models that can perceive and generate speech, supporting cross-modal instruction following a… | 30 | 1400 | active |