Open-source llm-inference software
3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
3358 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| EvanZhouDev/openai-oauth A TypeScript toolkit that turns a ChatGPT account into an OpenAI-compatible API via a local dev proxy, TypeScript SDK, and client adapters.… | 66 | 1398 | active |
| explosion/spacy-llm spacy-llm is a Python library that integrates Large Language Models into spaCy NLP pipelines via a serializable llm component. It provides … | 75 | 1395 | active |
| aws-samples/generative-ai-use-cases Generative AI Use Cases (GenU) is a well-architected sample application from AWS that demonstrates business use cases for generative AI, bu… | 90 | 1390 | active |
| foxhui/WebAI2API WebAI2API is a self-hosted Node.js service that exposes web-based AI services (LMArena, Gemini, ChatGPT, DeepSeek, etc.) as OpenAI-compatib… | 54 | 1388 | active |
| meridianlabs-ai/inspect_petri Inspect Petri is an alignment auditing agent that automatically probes language models for concerning behaviors like sycophancy, reward hac… | 81 | 1385 | active |
| Zefan-Cai/KVCache-Factory KVCache-Factory is a unified Python framework implementing many KV cache compression, retrieval, merging, and quantization methods for long… | 65 | 1384 | active |
| lamm-mit/PDF2Audio A Gradio-based web application that converts PDF documents into audio podcasts, lectures, and summaries using OpenAI GPT models for text ge… | 27 | 1384 | active |
| cccyd2003-qwq/pinkbin Pinkbin is an open-source Windows disk analyzer and cleaner built with Tauri 2, React, and Rust. It scans entire drives in seconds via dire… | 55 | 1383 | active |
| ali-vilab/TeaCache TeaCache is a training-free caching approach that accelerates inference for video diffusion models by estimating output differences across … | 30 | 1383 | active |
| elevenyellow/handcrafted-persona-engine Persona Engine is a Windows desktop application that drives a Live2D avatar with an AI pipeline: microphone speech recognition, an LLM guid… | 67 | 1381 | active |
| YuvDwi/Steve Steve is a Minecraft Forge mod that adds autonomous AI agents ('Cursor for Minecraft') that play the game alongside the player. Players giv… | 56 | 1381 | active |
| QwenAudio/ThinkSound ThinkSound is a PyTorch implementation of a NeurIPS 2025 framework that generates and edits audio from video, text, or audio inputs using C… | 49 | 1380 | active |
| 1rgs/jsonformer Jsonformer is a Python library that wraps Hugging Face causal language models to generate syntactically correct JSON conforming to a given … | 30 | 4936 | maintenance |
| felixrieseberg/clippy A desktop chat application that runs local LLMs (GGUF models via llama.cpp) behind a nostalgic 1990s Microsoft Office Clippy assistant inte… | 42 | 1378 | active |
| ServiceNow/BrowserGym BrowserGym is a Gym-style environment library for web task automation, providing a unified interface to web agent benchmarks like MiniWoB, … | 78 | 1376 | active |
| K-Dense-AI/k-dense-byok K-Dense BYOK is a free, open-source desktop application providing 'Kady', an AI research assistant (co-scientist) that runs locally and use… | 78 | 1376 | active |
| PentesterFlow/agent PentesterFlow is a terminal-based agentic AI CLI assistant for penetration testers and bug bounty hunters. It orchestrates LLM-driven recon… | 72 | 1375 | active |
| mpociot/chatgpt-vscode A Visual Studio Code extension that integrates ChatGPT into the editor, letting users ask questions, query selected code snippets, and inse… | 32 | 4920 | maintenance |
| Lapis0x0/obsidian-yolo YOLO is an agent-native AI assistant plugin for Obsidian that combines chat, writing assistance, knowledge base interaction, and agent orch… | 83 | 1372 | active |
| andimarafioti/faster-qwen3-tts A Python library for real-time text-to-speech inference with Qwen3-TTS using manual CUDA graph capture, requiring no Flash Attention, vLLM,… | 82 | 1372 | active |
| ChatWise ChatWise is a fast, privacy-friendly desktop AI chatbot client that connects to multiple LLM providers (OpenAI, Anthropic, Google, Groq, Ol… | 67 | 1371 | active |
| Meituan-AutoML/MobileVLM MobileVLM is a family of compact vision language models (1.4B-3B parameters) designed to run efficiently on mobile devices, combining small… | 17 | 1371 | active |
| JuneYaooo/gpt-image2-ppt-skills A Claude Code / OpenClaw skill that generates polished 16:9 presentations using OpenAI's gpt-image-2, rendering each slide as a complete vi… | 55 | 1370 | active |
| HazyResearch/minions Minions is a communication protocol and Python framework that lets small on-device LLMs collaborate with frontier cloud LLMs to cut cloud i… | 49 | 1370 | active |
| OpenRaiser/NanoResearch NanoResearch is an autonomous AI research assistant that runs an end-to-end pipeline from a research idea to a complete LaTeX paper. It exe… | 57 | 1369 | active |
| Kuingsmile/word-GPT-Plus Word GPT Plus is a Microsoft Word add-in that integrates LLMs and AI agents directly into Word for generating, translating, summarizing, an… | 83 | 1368 | active |
| OpenPPL/ppl.nn PPLNN is a high-performance deep-learning inference engine written in C++ that runs ONNX models on x86 CPUs and NVIDIA GPUs, with a dedicat… | 32 | 1368 | active |
| NVIDIA-AI-IOT/torch2trt torch2trt is a Python library that converts PyTorch models to TensorRT engines using the TensorRT Python API, with a simple single-function… | 23 | 4885 | maintenance |
| ntegrals/10x 10x is an open-source terminal-based AI coding agent that chains multiple LLM models through multi-step 'Superpowers' workflows with smart … | 43 | 1366 | active |
| ThousandBirdsInc/chidori Chidori is an agent framework that runs async TypeScript agents on an embedded pure-Rust JavaScript engine, recording every side effect (LL… | 92 | 1365 | active |
| alibaba/rtp-llm RTP-LLM is Alibaba's high-performance LLM inference engine written in C++/CUDA, optimized with kernels like PagedAttention and FlashAttenti… | 65 | 1364 | active |
| raznem/parsera Parsera is a lightweight Python library for scraping websites using LLMs, letting users define elements to extract with natural-language de… | 49 | 1363 | active |
| FreedomIntelligence/HuatuoGPT-o1 HuatuoGPT-o1 is a medical large language model project for complex medical reasoning, with open-sourced models (7B-72B), fine-tuning data, … | 22 | 1363 | active |
| mishl-dev/Discord-AI-Chatbot A self-hosted Discord AI chatbot application written in Python that uses the Groq API for fast LLM responses. It supports slash and text co… | 10 | 1363 | active |
| wrtnlabs/autobe AutoBE is an AI-powered backend builder that turns natural language requirements into complete NestJS/TypeScript backend applications with … | 65 | 1361 | active |
| fastclaw-ai/fastclaw FastClaw is a lightweight AI agent runtime written in Go, distributed as a single binary. It acts as an agent factory that creates, manages… | 77 | 1359 | active |
| BillHuang2001/evogit EvoGit is a decentralized multi-agent framework that treats software development as an evolutionary process, deploying a population of inde… | 64 | 1358 | active |
| MoonshotAI/Kimi-Dev Kimi-Dev-72B is an open-source coding LLM from Moonshot AI specialized in software engineering tasks like issue resolution and bug fixing, … | 35 | 1355 | active |
| vercel-labs/gemini-chatbot An open-source Next.js chatbot template powered by Google Gemini and the Vercel AI SDK, with streaming chat, generative UI, chat history pe… | 60 | 1354 | active |
| Abraxas-365/langchain-rust LangChain for Rust is the Rust language implementation of the LangChain framework for building LLM-powered applications through composabili… | 94 | 1350 | active |
| astaxie/TokenHub TokenHub is a self-hosted enterprise AI gateway written in Go that provides a governance layer over AI model access, including model routin… | 79 | 1350 | active |
| sashiko-dev/sashiko Sashiko is an agentic code review system for the Linux kernel that ingests patches from mailing lists, GitHub PRs, GitLab MRs, or local git… | 61 | 1346 | active |
| LoLLMs LoLLMs WebUI is a local, single-user web interface for running large language models and multimodal AI systems, supporting hundreds of mode… | 64 | 4789 | maintenance |
| hao-ai-lab/LookaheadDecoding A Python library implementing Lookahead Decoding, an exact parallel decoding algorithm that accelerates LLM inference without a draft model… | 28 | 1343 | active |
| louisho5/picobot Picobot is a lightweight, self-hosted AI agent distributed as a single ~9MB Go binary with persistent memory, tool calling, skills, and mes… | 80 | 1342 | active |
| inference-labs-inc/dsperse DSperse is a proving-system-agnostic tool for verifiable AI that decomposes ONNX neural network models into circuit-compatible segments and… | 76 | 1342 | active |
| jbilcke-hf/ai-comic-factory AI Comic Factory is a web application that generates comic panels and pages from a single text prompt by combining an LLM for story/dialogu… | 10 | 1342 | active |
| giuseppe99barchetta/SuggestArr SuggestArr is a self-hosted web application that automatically recommends and requests movies, TV shows, and anime to Jellyseerr/Overseerr … | 89 | 1341 | active |
| 1692775560/deepseek_project A collection of DeepSeek-related tools including a WeChat chatbot that uses the DeepSeek API for automated replies, a local document upload… | 63 | 1341 | active |
| stenolabs/stenoai Steno is a privacy-first desktop AI notepad and meeting notetaker that records, transcribes, summarizes, and lets you query meetings entire… | 85 | 1338 | active |
| ChaokunHong/MetaScreener MetaScreener is an open-source AI tool that automates title/abstract and full-text PDF screening for systematic reviews using an ensemble o… | 65 | 1336 | active |
| lean-dojo/LeanCopilot Lean Copilot integrates large language models natively into the Lean 4 theorem prover for proof automation. It provides tactic suggestion, … | 91 | 1335 | active |
| bevibing/tutor-skills A pair of Claude Code skills that convert documents (PDF, MD, HTML, EPUB) or source codebases into structured Obsidian study vaults with in… | 43 | 1333 | active |
| facebookincubator/AITemplate AITemplate is a Python framework that compiles deep neural networks into high-performance CUDA (NVIDIA) or HIP (AMD) C++ code for fast fp16… | 67 | 4730 | maintenance |
| Vahe1994/AQLM Official PyTorch implementation of AQLM, an extreme LLM compression method via additive quantization, extended with PV-Tuning for finetunin… | 54 | 1329 | active |
| yukkcat/gemini-business2api A self-hosted gateway service that exposes Gemini Business through an OpenAI-compatible API, with multi-account load balancing and an admin… | 52 | 1328 | active |
| apple/python-apple-fm-sdk Python bindings for Apple's Foundation Models framework, giving access to the on-device foundation model behind Apple Intelligence on macOS… | 76 | 1327 | active |
| Tongjilibo/bert4torch bert4torch is a PyTorch library providing an elegant reimplementation of transformer models (BERT, RoBERTa, T5, GPT, ChatGLM, LLaMA, etc.) … | 76 | 1326 | active |
| AutumnWhj/ChatGPT-wechat-bot A WeChat bot that connects to ChatGPT/OpenAI's API so your WeChat account can reply to messages with AI-generated answers. It supports priv… | 50 | 4715 | maintenance |
| SSShooter/ebook-to-mindmap A browser-based AI application that parses EPUB and PDF ebooks and converts them into chapter summaries, per-chapter mind maps, or a single… | 68 | 1325 | active |
| Duxiaoman-DI/XuanYuan XuanYuan is a family of open-source Chinese financial-domain large language models from Duxiaoman, including base, chat, and quantized vari… | 30 | 1324 | active |
| xunbu/docutranslate DocuTranslate is a lightweight local document translation tool powered by large language models, supporting formats such as pdf, docx, xlsx… | 78 | 1322 | active |
| meta-pytorch/segment-anything-fast A fast, batched offline inference-oriented fork of Meta's Segment Anything (SAM) image segmentation model. It applies optimizations like bf… | 42 | 1321 | active |
| Robitx/gp.nvim Gp.nvim is a Neovim plugin that brings GPT-powered chat sessions, instructable text/code operations, speech-to-text, and image generation d… | 33 | 1320 | active |
| Capsize-Games/airunner AI Runner is a privacy-focused desktop application for running local AI models offline, combining an AI chat companion with voice conversat… | 92 | 1319 | active |
| ConardLi/easy-llm-cli Easy LLM CLI is an open-source command-line AI agent forked from Gemini CLI that works with multiple LLM providers, including Gemini, OpenA… | 45 | 1319 | active |
| evilsocket/nerve Nerve is a simple Agent Development Kit (ADK) for building, running, evaluating, and orchestrating LLM-based agents using YAML definitions … | 10 | 1318 | active |
| run-bigpig/jcp JCP (韭菜盘) is a cross-platform desktop application built with Wails, Go, and React that provides AI-driven analysis of Chinese A-share stock… | 10 | 1318 | active |
| sanchit-gandhi/whisper-jax An optimized JAX implementation of OpenAI's Whisper speech recognition model, built on Hugging Face Transformers, offering up to 70x faster… | 31 | 4680 | maintenance |
| tsinghua-fib-lab/AgentSociety AgentSociety is an LLM-native agent simulation framework for building large-scale multi-agent simulations of social and urban environments.… | 86 | 1317 | active |
| SanMuzZzZz/LuaN1aoAgent LuaN1aoAgent is an autonomous AI-driven penetration testing agent built in TypeScript on the Pi SDK, using graph-based cognitive reasoning … | 77 | 1316 | active |
| HugeCatLab/ChatTutor ChatTutor is a visual and interactive AI tutoring application that gives LLMs the ability to teach on an electronic whiteboard. It combines… | 49 | 1315 | active |
| huggingface/llm-vscode A VSCode extension providing LLM-powered features like Copilot-style ghost-text code completion, with configurable backends including the H… | 65 | 1314 | active |
| neurogen-dev/NeuroAPI NeuroAPI is a Russian AI API gateway providing unified OpenAI/Anthropic/Gemini-compatible access to models like GPT, Claude, Gemini, and De… | 63 | 1313 | active |
| tjake/Jlama Jlama is a modern LLM inference engine written in Java, supporting popular transformer models like Llama, Mistral, Gemma, and Qwen with qua… | 48 | 1309 | active |
| aws-samples/claude-prompt-generator A Python web application that generates, translates, and iteratively evaluates prompts for Anthropic Claude 3 via AWS Bedrock, including co… | 10 | 1309 | active |
| openai/simple-evals A lightweight Python library from OpenAI for evaluating language models against benchmarks like MMLU, GPQA, MATH, HumanEval, SimpleQA, Heal… | 57 | 4638 | maintenance |
| SakanaAI/text-to-lora Text-to-LoRA (T2L) is a hypernetwork that generates LoRA adapters for large language models in a single forward pass, using only a natural … | 28 | 1307 | active |
| DimiMikadze/orca Orca is an AI agent application for deep LinkedIn profile analysis that scrapes posts, comments, reactions, and interaction networks, then … | 76 | 1306 | active |
| DAMO-NLP-SG/VideoLLaMA2 VideoLLaMA 2 is an open-source video large language model that adds spatial-temporal modeling and audio understanding to video-LLMs. It pro… | 25 | 1306 | active |
| turboderp-org/exllamav2 ExLlamaV2 is a fast Python inference library for running large language models locally on modern consumer GPUs, with support for EXL2/GPTQ … | 58 | 4631 | maintenance |
| dabit3/react-native-ai React Native AI is a full-stack framework for building cross-platform mobile AI apps with React Native and an Express server proxy. It prov… | 70 | 1304 | active |
| neo4j/neo4j-graphrag-python The official Neo4j first-party Python library for building graph retrieval-augmented generation (GraphRAG) applications. It provides retrie… | 92 | 1303 | active |
| datalayer/jupyter-mcp-server A Model Context Protocol (MCP) server that lets AI agents connect to, edit, and execute Jupyter Notebooks in real time. It supports STDIO a… | 82 | 1300 | active |
| coleam00/claude-memory-compiler A Python tool that gives Claude Code an evolving memory by capturing conversation transcripts via hooks, extracting decisions and lessons w… | 45 | 1299 | active |
| lasantosr/intelli-shell IntelliShell is a command template and snippet manager for shells, offering smart search, dynamic variable completions, bookmarks, and AI-p… | 98 | 1296 | active |
| s2b-dev/smart-second-brain An open-source Obsidian plugin that adds a privacy-focused AI assistant to your note vault. It embeds your notes and uses a RAG pipeline wi… | 92 | 1296 | active |
| Codeium/Windsurf A native Neovim plugin for Windsurf (formerly Codeium) that provides AI-powered code completion and chat inside the editor. It integrates w… | 62 | 1295 | active |
| rinadelph/Agent-MCP Agent-MCP is a TypeScript framework for building coordinated multi-agent AI systems over the Model Context Protocol (MCP). It provides a pe… | 51 | 1295 | active |
| ngxson/wllama wllama is a WebAssembly binding for llama.cpp that enables running LLM inference entirely in the browser, with WebGPU support, multimodal i… | 89 | 1294 | active |
| Xiangyue-Zhang/auto-deep-researcher-24x7 An open-source Python framework where LLM agents autonomously run deep learning experiments 24/7, covering hypothesis formation, code imple… | 50 | 1293 | active |
| panyanyany/Twocast Twocast is a self-hostable AI podcast generator that creates two-person conversational podcast episodes from topics, links, documents, or w… | 29 | 1292 | active |
| ELLA ELLA is an Efficient Large Language Model Adapter that equips text-to-image diffusion models with LLM-based text understanding via a Timest… | 26 | 1289 | active |
| PrunaAI/pruna Pruna is an open-source Python model optimization framework that makes AI models faster, smaller, cheaper, and greener via caching, quantiz… | 80 | 1288 | active |
| vipshop/cache-dit Cache-DiT is a PyTorch-native inference engine that accelerates Diffusion Transformer (DiT) models with hybrid caching, parallelism, quanti… | 79 | 1287 | active |
| joeseesun/qiaomu-mondo-poster-design An AI-powered poster and cover design tool that generates professional-grade posters, book covers, album covers, and social media graphics … | 44 | 1287 | active |
| Muesli-HQ/muesli Muesli is an open-source native macOS app that combines hotkey-driven AI dictation with local meeting transcription, running speech-to-text… | 79 | 1285 | active |
| DropbaseHQ/dropbase Dropbase is a local-first, self-hosted platform that uses AI to generate Python web apps, combining a drag-and-drop UI builder with editabl… | 27 | 1285 | active |
| kubeai-project/kubeai KubeAI is a Kubernetes operator for serving machine learning models in production, supporting LLMs via vLLM and Ollama, vector embeddings, … | 95 | 1284 | active |
| poetiq-ai/poetiq-arc-agi-solver A Python research codebase that reproduces Poetiq's record-breaking, leaderboard-topping submissions to the ARC-AGI-1 and ARC-AGI-2 abstrac… | 39 | 1283 | active |