Open-source llm-inference software
3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
3358 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| Yuyz0112/claude-code-reverse A tool that reverse engineers Claude Code by capturing and visualizing its LLM API interactions. It monkey-patches Claude Code's request co… | 34 | 2432 | active |
| jackmpcollins/magentic A Python library for seamlessly integrating LLMs into Python code via @prompt and @chatprompt decorators that return typed, structured outp… | 69 | 2428 | active |
| microsoft/chat-copilot Chat Copilot is a sample application from Microsoft for building your own LLM-powered chat copilot, built on Semantic Kernel. It consists o… | 10 | 2428 | active |
| trymirai/uzu Uzu is a high-performance inference engine written in Rust for running AI models directly on-device, with Python, TypeScript, and Swift bin… | 87 | 2423 | active |
| Prism Prism is a Laravel package providing a unified, fluent interface for working with Large Language Models across providers like OpenAI, Anthr… | 63 | 2423 | active |
| VoiceInput A macOS menu-bar voice input app (Swift, macOS 14+) that lets users hold the Fn key to record and release to inject streaming-transcribed t… | 47 | 2419 | active |
| zurawiki/gptcommit gptcommit is a git prepare-commit-msg hook written in Rust that uses large language models like GPT to automatically generate clear, descri… | 85 | 2418 | active |
| Jittor/JittorLLMs JittorLLMs is a large language model inference library built on the Jittor deep learning framework, supporting models like ChatGLM, LLaMA/L… | 31 | 2406 | active |
| BloopAI/bloop bloop is a fast code search engine and desktop app that lets developers search their codebase with natural language, regex, and symbol quer… | 10 | 9488 | maintenance |
| Olive Olive is Microsoft's AI model optimization toolkit for the ONNX Runtime, automating finetuning, conversion, quantization, and compression o… | 89 | 2402 | active |
| spacedriveapp/spacebot Spacebot is an opinionated AI agent harness written in Rust for teams, communities, and multi-user environments, deployable as a single sel… | 73 | 2400 | active |
| drl990114/MarkFlowy MarkFlowy is a lightweight, cross-platform Markdown and text editor built with Tauri, React, and Prosemirror, offering WYSIWYG and source-c… | 99 | 2398 | active |
| gkamradt/needle-in-a-haystack A benchmarking tool (niah / needlehaystack) that pressure-tests LLM long-context retrieval by sweeping context length x needle depth cells … | 66 | 2397 | active |
| yawiii/ComfyUI-Prompt-Assistant A ComfyUI plugin that provides an all-in-one prompt assistant, connecting to cloud LLM/VLM APIs (Zhipu, SiliconFlow, Gemini, Baidu) and loc… | 64 | 2394 | active |
| zai-org/GLM-V GLM-V is the open-source repository for Zhipu AI's GLM-4.6V, GLM-4.5V, and GLM-4.1V-Thinking vision-language models, which perform versatil… | 63 | 2393 | active |
| aiwithremy/claude-skills-llm-council A Claude Code skill that runs a user's question through five independent AI advisors who peer-review each other's answers, then synthesizes… | 47 | 2384 | active |
| DeepInsight-AI/DeepBI DeepBI is an AI-native business intelligence platform that uses large language models to let users explore, query, visualize, and share dat… | 59 | 2382 | active |
| showlab/Paper2Video Paper2Video is a Python pipeline that automatically generates academic presentation videos from scientific papers, taking a paper PDF, a sp… | 46 | 2381 | active |
| agentuniverse-ai/agentUniverse agentUniverse is a Python-based multi-agent framework built on large language models, providing extensible components for building individu… | 69 | 2374 | active |
| XiaomiMiMo/MiMo Xiaomi's MiMo is a 7B-parameter reasoning language model trained from pretraining through posttraining with reinforcement learning, release… | 28 | 2373 | active |
| FMInference/FlexLLMGen FlexLLMGen is a high-throughput generation engine for running large language models on a single GPU with limited memory. It uses IO-efficie… | 10 | 9346 | maintenance |
| TransWithAI/Faster-Whisper-TransWithAI-ChickenRice A high-performance audio/video transcription and translation application built on Faster Whisper, optimized for Japanese-to-Chinese transla… | 71 | 2366 | active |
| orhanerday/open-ai A community-maintained PHP SDK for accessing the OpenAI API, supporting GPT-3/GPT-4 chat completions, DALL-E image generation, and ChatGPT-… | 24 | 2363 | active |
| E2B An open-source AI agent that controls a secure cloud Linux desktop (via E2B Desktop Sandbox) using keyboard, mouse, and shell commands, pow… | 60 | 2359 | active |
| UnicomAI/wanwu Yuanjing Wanwu is China Unicom's enterprise-grade, multi-tenant AI agent development platform built in Go. It provides all-in-one tooling f… | 83 | 2358 | active |
| Karben233/bili-hardcore A Rust CLI tool that automatically answers Bilibili's 'hardcore member' daily quiz questions using LLM APIs. It also ships as an agent skil… | 85 | 2355 | active |
| GalTransl/GalTransl GalTransl is an automated translation tool for visual novels (galgames) that leverages large language models like GPT-4, Claude, Deepseek, … | 97 | 2353 | active |
| fruitbars/simple-one-api A single-binary Go gateway that unifies access to multiple LLM providers (Qianfan, iFlytek Spark, Tencent Hunyuan, MiniMax, DeepSeek, etc.)… | 90 | 2345 | active |
| maitrix-org/llm-reasoners LLM Reasoners is a Python library that provides a standardized modular framework for advanced LLM reasoning, implementing algorithms like M… | 28 | 2345 | active |
| copilot-emacs/copilot.el An unofficial Emacs plugin for GitHub Copilot that provides inline ghost-text completions, a chat interface, and Next Edit Suggestions. It … | 99 | 2330 | active |
| zakirkun/deep-eye Deep Eye is an AI-driven penetration testing CLI that orchestrates multiple LLM providers (OpenAI, Claude, Gemini, OLLAMA, Groq, and others… | 64 | 2330 | active |
| refuel-ai/autolabel Autolabel is a Python library for labeling, cleaning, and enriching text datasets using large language models like GPT-4, Claude, and open-… | 22 | 2330 | active |
| jwadow/kiro-gateway A self-hosted proxy API gateway for the Kiro API (Amazon Q Developer / AWS CodeWhisperer) that exposes Claude and other models through Open… | 59 | 2325 | active |
| PhialsBasement/Chain-of-Recursive-Thoughts CoRT (Chain of Recursive Thoughts) is a tool that makes LLMs recursively refine their answers by generating multiple alternative responses,… | 25 | 2325 | active |
| PKU-YuanGroup/MoE-LLaVA MoE-LLaVA is an open-source Mixture-of-Experts based sparse large vision-language model, released with the MoE-Tuning training strategy fro… | 28 | 2324 | active |
| severian42/GraphRAG-Local-UI A local-first GraphRAG application adapted from Microsoft's GraphRAG, providing a FastAPI backend plus Gradio-based UIs for indexing, promp… | 15 | 2320 | active |
| romgX/openrelay OpenRelay is a desktop application that aggregates AI model quotas from local tools (Claude Desktop, Kiro, Windsurf, Copilot, etc.) and dir… | 75 | 2307 | active |
| Neroued/ninfer NInfer is a from-scratch C++/CUDA inference engine optimized for maximum single-GPU performance on a narrow, explicitly registered set of Q… | 57 | 2305 | active |
| MarkPDFdown MarkPDFDown is a Python CLI tool that converts PDF documents and images into clean Markdown using multimodal large language models via Lite… | 72 | 2302 | active |
| rushindrasinha/youtube-shorts-pipeline Verticals v3 (repo youtube-shorts-pipeline) is a Python CLI that automates producing and publishing YouTube Shorts: it researches a topic, … | 55 | 2301 | active |
| yan5xu/ququ QuQu is an open-source, free desktop voice dictation app built with Electron, designed as a privacy-first Wispr Flow alternative optimized … | 25 | 2296 | active |
| Open-Curiosity/gini-agent Gini Agent is a personal AI agent runtime that remembers conversations, learns skills from task outcomes, and exposes chat, CLI, web, and m… | 69 | 2294 | active |
| OpenChatKit OpenChatKit is an open-source toolkit from Together, LAION, and Ontocord.ai for building and fine-tuning chat language models, including in… | 31 | 8983 | maintenance |
| jamubc/gemini-mcp-tool A Model Context Protocol (MCP) server that lets AI assistants like Claude Code invoke the Google Gemini CLI (or its successor, the Antigrav… | 73 | 2290 | active |
| HKUDS/FastCode FastCode is a token-efficient framework for repository-scale code understanding and reasoning, using structural scouting over a semantic ma… | 52 | 2289 | active |
| 777genius/agent-teams-ai A free Electron desktop app for orchestrating teams of AI agents that handle tasks autonomously, message each other, and review each other'… | 83 | 2288 | active |
| InternLM/lagent Lagent is a lightweight Python framework for building LLM-based agents, designed with a PyTorch-like philosophy where agents are composed l… | 80 | 2283 | active |
| MCPJam/inspector MCPJam is an open-source testing and evaluation platform for MCP server developers, providing an inspector playground, OAuth/EMA debugger, … | 83 | 2274 | active |
| NVIDIA-NeMo/labs-OO-Agents NVIDIA Object Oriented Agents (NOOA) is a model-agnostic Python framework for building AI agents where state, tools, prompts, and typed int… | 78 | 2273 | active |
| baturyilmaz/wordpecker-app WordPecker is a personalized language-learning web application that combines Duolingo-style lessons with user-curated vocabulary lists. It … | 33 | 2272 | active |
| aixcoder-plugin/aiXcoder-7B Official repository for aiXcoder-7B, an open-weights 7B-parameter code large language model trained on 1.2T tokens for code completion, gen… | 26 | 2271 | active |
| getkimchi/kimchi Kimchi is a terminal-based AI coding agent CLI that autonomously executes multi-step development tasks like refactoring, writing tests, and… | 78 | 2268 | active |
| Fullmoon fullmoon is a free, open-source iOS/iPadOS/macOS/visionOS app for chatting with local large language models, built on Apple's MLX Swift fra… | 30 | 2268 | active |
| qhjqhj00/MemoRAG MemoRAG is a Python RAG framework that uses a super-long memory model to build a global understanding of an entire corpus (up to ~1M tokens… | 29 | 2266 | active |
| moorcheh-ai/memanto Memanto is a companion 'memory agent' that manages long-term semantic memory for AI agents — deciding what to keep, resolving contradiction… | 79 | 2261 | active |
| numman-ali/cc-mirror CC-MIRROR is an npm-distributed CLI that creates isolated, provider-native variants of Claude Code wired to alternative LLM providers such … | 66 | 2261 | active |
| aattaran/deepclaude A shell/PowerScript wrapper that launches Claude Code's autonomous coding agent loop backed by cheaper Anthropic-compatible LLM providers l… | 53 | 2261 | active |
| antoinezambelli/forge Forge is a Python framework providing a reliability layer for self-hosted LLM tool-calling, with guardrails like rescue parsing, retry nudg… | 80 | 2257 | active |
| apconw/Aix-DB Aix-DB is an AI-powered data analysis system (ChatBI) built on LangChain/LangGraph with an MCP Skills multi-agent architecture, converting … | 74 | 2254 | active |
| OpenSPG/openspg OpenSPG is a knowledge graph engine built on the SPG (Semantic-enhanced Programmable Graph) framework, developed by Ant Group with OpenKG. … | 35 | 2252 | active |
| tphakala/birdnet-go BirdNET-Go is a self-hosted, 24/7 realtime soundscape analyser that classifies birds, bats, and other wildlife using local AI models like B… | 92 | 2250 | active |
| Ocyss/boss-helper A browser extension (also available as a userscript) that enhances the Boss Zhipin job platform by removing ads, improving the UI, enabling… | 85 | 2249 | active |
| tsingyuai/scientify Scientify is an OpenClaw plugin that automates end-to-end scientific research workflows, continuously ingesting new papers, evolving hypoth… | 71 | 2244 | active |
| datapizza-labs/datapizza-ai Datapizza AI is a Python framework for building production-ready generative AI applications with agents, LLM clients, and RAG pipelines. It… | 63 | 2243 | active |
| microsoft/LLaVA-Med LLaVA-Med is a large language-and-vision assistant fine-tuned for the biomedicine domain, built on the LLaVA multimodal architecture. It su… | 37 | 2237 | active |
| Doriandarko/deepseek-engineer DeepSeek Engineer is a Python command-line coding assistant that connects to the DeepSeek API and uses function calling to read, create, an… | 29 | 2231 | active |
| tnm/zclaw zclaw is a personal AI assistant firmware written in C that runs entirely on ESP32 microcontrollers within a strict 888 KiB all-in firmware… | 60 | 2230 | active |
| floneum/kalosm Kalosm is a Rust ecosystem of crates providing simple interfaces for running pre-trained language, audio, and image models locally or remot… | 65 | 2227 | active |
| espressif/esp-claw ESP-Claw is Espressif's 'Chat Coding' AI agent framework for IoT devices, reimplemented in C for ESP32-series chips. It lets users define d… | 64 | 2227 | active |
| AnotiaWang/deep-research-web-ui A web UI for the dzhng/deep-research project that performs iterative, AI-driven research by combining search engines, web scraping, and LLM… | 88 | 2223 | active |
| ai-christianson/RA.Aid RA.Aid is a standalone autonomous coding agent built on LangGraph that researches, plans, and implements multi-step software development ta… | 49 | 2222 | active |
| op7418/Youtube-clipper-skill A Claude Code skill that downloads YouTube videos and uses AI to generate semantic chapters, clip segments, translate subtitles into biling… | 41 | 2217 | active |
| langchain-ai/executive-ai-assistant Executive AI Assistant (EAIA) is an open-source AI agent built with LangGraph that performs the job of an executive assistant, primarily by… | 10 | 2207 | active |
| Core AI Apple's repository of model export recipes, Python primitives, and Swift runtime utilities for building on-device AI with the Core AI frame… | 70 | 2206 | active |
| NX-AI/xlstm Official PyTorch implementation of xLSTM, an extended Long Short-Term Memory recurrent architecture with exponential gating and matrix memo… | 76 | 2205 | active |
| mozilla-ai/any-llm any-llm is a Python library from Mozilla.ai that provides a single, unified interface for calling many LLM providers (OpenAI, Anthropic, Mi… | 78 | 2204 | active |
| numman-ali/opencode-openai-codex-auth An OpenCode plugin that authenticates via OpenAI's official ChatGPT OAuth flow, letting users with ChatGPT Plus/Pro subscriptions access GP… | 49 | 2203 | active |
| MiniMax AI Platform The official CLI for the MiniMax AI Platform, written in TypeScript, that generates text, images, video, speech, and music from the termina… | 83 | 2202 | active |
| RightNow-AI/picolm PicoLM is a pure C11 LLM inference engine that runs 1-billion parameter models in GGUF format on extremely constrained hardware like $10 bo… | 43 | 2202 | active |
| MoonshotAI/MoBA MoBA (Mixture of Block Attention) is a PyTorch implementation of a trainable block-sparse attention mechanism for long-context large langua… | 24 | 2191 | active |
| TimmyOVO/deepseek-ocr.rs A Rust implementation of the DeepSeek-OCR inference stack with multiple OCR/VLM backends (DeepSeek-OCR, PaddleOCR-VL, DotsOCR), DSQ quantiz… | 54 | 2189 | active |
| Jazee6/cloudflare-ai-web A self-hosted web application for chatting with multiple AI models via Cloudflare Workers AI and AI Gateway (including Gemini). Built with … | 94 | 2188 | active |
| withcatai/node-llama-cpp node-llama-cpp is a Node.js library providing bindings to llama.cpp for running LLMs locally, with pre-built binaries and automatic GPU sup… | 95 | 2187 | active |
| ggml-org/llama.vim A Vim/Neovim plugin that provides LLM-assisted code and text completion via a local llama.cpp server. It offers fill-in-the-middle (FIM) au… | 75 | 2187 | active |
| pretzelai/pretzelai Pretzel is an open-source fork of Jupyter Notebooks that adds AI code generation, inline tab completion, sidebar chat, and error fixing whi… | 16 | 2186 | active |
| hls4ml hls4ml is a Python package that converts machine learning models from Keras, PyTorch, and ONNX into high-level synthesis (C++) code for FPG… | 80 | 2185 | active |
| rosemarycox5334-debug/PA_Agent A desktop AI-assisted Price Action analysis tool for discretionary traders. It ingests candlestick data from MT5, TradingView, yfinance, or… | 59 | 2180 | active |
| intel/intel-extension-for-transformers Intel's toolkit for accelerating transformer-based GenAI/LLM workloads on Intel platforms, offering state-of-the-art compression (e.g., INT… | 10 | 2169 | active |
| Deep Code Deep Code CLI is an open-source terminal-based AI coding assistant optimized for DeepSeek-V4 models, supporting deep thinking mode, reasoni… | 81 | 2165 | active |
| warden-protocol/wardenprotocol Warden Protocol is a blockchain protocol monorepo providing a network layer for AI agents, including an agentic wallet, a peer-to-peer perm… | 87 | 2161 | active |
| boson-ai/higgs-audio Higgs Audio is a text-audio foundation model project from Boson AI providing code and weights for conversational text-to-speech with zero-s… | 53 | 8369 | maintenance |
| rupeshs/fastsdcpu FastSD CPU is a Python application that runs Stable Diffusion image generation quickly on CPUs and Intel AI PCs using Latent Consistency Mo… | 75 | 2158 | active |
| xuanyustudio/LocalMiniDrama LocalMiniDrama is an open-source Electron desktop application that generates AI short dramas and animated dramas end-to-end, from story and… | 83 | 2157 | active |
| Nutlope/notesGPT NotesGPT is an open-source AI-powered voice note-taking web app that records voice notes, transcribes them with Whisper, and generates summ… | 65 | 2155 | active |
| Alisa0808/vox-director An agent skill that turns a single topic into a finished Vox-style paper-collage explainer or ad video, automating script, keyframes, motio… | 58 | 2151 | active |
| rynfar/meridian Meridian is a proxy that bridges Anthropic's Claude Agent SDK to the standard Anthropic API format, letting a Claude Max subscription power… | 80 | 2148 | active |
| chi111i/BypassAIGC A self-hosted AI academic writing assistant that polishes and rewrites papers using LLM APIs (e.g., Gemini via OpenAI-compatible endpoints)… | 78 | 2145 | active |
| btahir/open-deep-research An open-source web application that replicates Gemini's Deep Research by searching the web, extracting page content, and generating AI-writ… | 10 | 2142 | active |
| anthropics/anthropic-sdk-typescript The official TypeScript/JavaScript SDK for accessing Anthropic's Claude language model APIs from server-side applications. It provides a ty… | 93 | 2132 | active |
| PKU-YuanGroup/LLaVA-CoT LLaVA-CoT is an 11B vision language model with training, inference, and dataset-generation code for spontaneous step-by-step multimodal rea… | 44 | 2131 | active |