Open-source llm-inference software
3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
3358 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| Nutlope/llamacoder Llama Coder is an open-source web application that generates small apps from a single prompt, similar to Claude Artifacts, powered by Llama… | 67 | 7130 | active |
| ymcui/Chinese-LLaMA-Alpaca-2 A project releasing Chinese-adapted LLaMA-2 base models and Alpaca-2 instruction-tuned chat models with an extended Chinese vocabulary, inc… | 51 | 7112 | stable |
| chenfei-wu/TaskMatrix TaskMatrix (Visual ChatGPT) is a Python framework that connects ChatGPT with a suite of visual foundation models like Stable Diffusion, Gro… | 31 | 33957 | maintenance |
| albertan017/LLM4Decompile LLM4Decompile is an open-source series of large language models (1.3B to 33B) trained to decompile binary code back into readable, executab… | 53 | 7076 | active |
| opensquilla/opensquilla OpenSquilla is a token-efficient, microkernel AI agent that routes each conversation turn to the cheapest capable LLM via a local model rou… | 80 | 7069 | active |
| zai-org/GLM-4 Official repository for the GLM-4 series of open-weight, multilingual (primarily Chinese/English) multimodal chat language models from Z.ai… | 64 | 7069 | active |
| google/gemma.cpp A lightweight, standalone C++ inference engine for Google's Gemma foundation models (Gemma 2/3, PaliGemma 2), with a small ~2K LoC core and… | 73 | 7060 | active |
| TheTom/turboquant_plus TurboQuant+ is a Python reference implementation of the TurboQuant KV cache compression method (ICLR 2026), using PolarQuant codebooks and … | 54 | 7036 | active |
| antvis/Infographic AntV Infographic is a declarative infographic generation and rendering engine for the web, with a custom syntax, ~200 built-in templates, t… | 77 | 7018 | active |
| deepseek-ai/DeepSeek-Coder-V2 DeepSeek-Coder-V2 is an open-weight Mixture-of-Experts code language model family released by DeepSeek AI, with weights available on Huggin… | 44 | 7012 | active |
| dsh-routing-suite A bundled kit for the DSH (DeepSeek harness) agent runtime combining a runtime injector plugin with task-aware reasoning-mode router preset… | 76 | 6999 | active |
| mckaywrigley/chatbot-ui Chatbot UI is an open-source, self-hostable web chat application for conversing with any AI model, with chat history and multi-model suppor… | 31 | 33349 | maintenance |
| open-multi-agent/open-multi-agent Open Multi-Agent (OMA) is an MIT-licensed TypeScript framework for multi-agent AI orchestration that runs inside Node.js backends. A coordi… | 81 | 6974 | active |
| CapsWriter-Offline CapsWriter-Offline is a fully offline voice input tool for Windows that transcribes speech to text when you hold CapsLock or mouse side but… | 88 | 6955 | active |
| mishushakov/llm-scraper A TypeScript library that turns any webpage into structured data using LLMs, built on Playwright and the Vercel AI SDK. It supports multipl… | 68 | 6937 | active |
| SenseNova SenseNova-U is a series of open-weight unified multimodal models (e.g., SenseNova-U1.5-8B-MoT) built on the NEO-unify architecture that com… | 58 | 6884 | active |
| dataease/SQLBot SQLBot is an open-source conversational data analysis (ChatBI) system that converts natural language questions into SQL queries using large… | 86 | 6880 | active |
| drumih/turbo-fieldfare TurboFieldfare is a custom Swift + Metal runtime that runs Gemma 4 26B-A4B inference in roughly 2 GB of RAM on Apple Silicon Macs by stream… | 79 | 6880 | active |
| olimorris/codecompanion.nvim CodeCompanion.nvim is a Neovim plugin that integrates LLMs (Anthropic, OpenAI, Gemini, Copilot, Ollama, and more) and CLI coding agents (Cl… | 87 | 6872 | active |
| kvcache-ai/Mooncake Mooncake is a KVCache-centric disaggregated serving platform for LLM inference, originally built to serve Kimi by Moonshot AI. It separates… | 90 | 6835 | active |
| stagewise-io/stagewise Stagewise is an open-source agentic IDE with a built-in coding agent that can see live app context, run git workflows, and orchestrate mult… | 80 | 6823 | active |
| postgresml/postgresml PostgresML is a PostgreSQL extension that brings machine learning and AI inference directly into the database, with GPU acceleration and pg… | 32 | 6823 | active |
| purocean/yn Yank Note is a highly extensible desktop Markdown editor built on Electron and the Monaco editor, with features like version control, docum… | 96 | 6760 | active |
| superagent-ai/superagent Superagent is an open-source SDK and platform for securing AI applications and agents, offering runtime guardrails that block prompt inject… | 73 | 6759 | active |
| kijai/ComfyUI-WanVideoWrapper ComfyUI custom nodes wrapping WanVideo (Wan2.1) and related video generation models. It provides a standalone sandbox for quickly implement… | 55 | 6737 | active |
| flashinfer-ai/flashinfer FlashInfer is a GPU kernel library and kernel generator for LLM inference, providing unified APIs for attention, GEMM, and MoE operations w… | 92 | 6728 | active |
| Forget-C/Jellyfish Jellyfish is a self-hosted, end-to-end production workspace for AI-generated short dramas, covering script input, storyboard breakdown, cha… | 68 | 6727 | active |
| Andyyyy64/whichllm whichllm is a Python CLI tool that auto-detects your GPU, CPU, and RAM, then ranks local LLMs from HuggingFace that will actually run and p… | 82 | 6724 | active |
| google-ai-edge/LiteRT-LM LiteRT-LM is Google's production-ready, high-performance open-source framework for running large language models on edge devices, built as … | 88 | 6702 | active |
| Beingpax/VoiceInk VoiceInk is a native macOS voice-to-text dictation app that transcribes speech to text almost instantly using local AI models (Parakeet, Wh… | 84 | 6693 | active |
| microsoft/LLMLingua LLMLingua is a Microsoft library for prompt compression that removes non-essential tokens from prompts and KV caches to speed up LLM infere… | 62 | 6687 | active |
| simplescaling/s1 s1 is an open-source research project implementing simple test-time scaling for large language models, including the s1K dataset of 1,000 c… | 30 | 6675 | active |
| BlockRunAI/ClawRouter ClawRouter is an open-source TypeScript LLM router built for autonomous AI agents, offering 71 models from OpenAI, Anthropic, Google, xAI, … | 79 | 6621 | active |
| ningzimu/codex-ppt-skill An agent skill (SKILL.md-based) that turns articles, reports, papers, or notes into image-based PowerPoint presentations by planning an out… | 80 | 6616 | active |
| julep-ai/julep Julep is an open-source framework and platform for building durable, composable AI agents as dataflows that can crash, resume, retry, and e… | 65 | 6576 | active |
| souzatharsis/podcastfy Podcastfy is an open-source Python package and CLI that transforms multimodal content (websites, PDFs, images, YouTube videos, topics) into… | 57 | 6576 | active |
| microsoft/call-center-ai An AI-powered call center service that lets you initiate or receive phone calls handled by an LLM-driven agent via a simple API call. Built… | 65 | 6570 | active |
| basketikun/chatgpt2api A self-hosted reverse-engineered implementation of ChatGPT's official web interfaces, exposing OpenAI-compatible API endpoints for text gen… | 71 | 6544 | active |
| shibing624/pycorrector pycorrector is a Python toolkit for Chinese text error correction, supporting phonetic, visual, and grammar error types with models includi… | 78 | 6528 | active |
| Genkit Genkit is an open-source framework by Google for building full-stack, AI-powered agentic applications in JavaScript/TypeScript, Go, Python,… | 87 | 6524 | stable |
| josStorer/RWKV-Runner RWKV-Runner is a lightweight desktop application that automates downloading, managing, and running RWKV large language models with a GUI. I… | 85 | 6486 | active |
| apfel apfel is a Swift CLI tool that exposes Apple's built-in on-device Foundation Models LLM on Apple Silicon Macs as a UNIX-friendly command, a… | 77 | 6453 | active |
| AIOS AIOS is an 'AI Agent Operating System' that embeds LLMs into an OS-like kernel providing scheduling, context switching, memory, storage, an… | 69 | 6445 | active |
| haifengl/smile SMILE is a comprehensive, high-performance machine learning framework for the JVM with idiomatic APIs for Java, Scala, and Kotlin. It cover… | 100 | 6423 | active |
| lavague-ai/LaVague LaVague is an open-source Python framework for building AI Web Agents powered by a Large Action Model. It combines a World Model that inter… | 26 | 6392 | active |
| E2B An open-source Next.js template for building AI-generated apps, similar to Claude Artifacts or v0. It uses the E2B SDK to securely execute … | 68 | 6388 | active |
| tensorflow/serving TensorFlow Serving is a flexible, high-performance serving system for machine learning models designed for production environments. It mana… | 89 | 6364 | stable |
| canopyai/Orpheus-TTS Orpheus TTS is an open-source text-to-speech system built on a Llama-3b backbone that produces human-sounding speech with emotion control a… | 42 | 6347 | active |
| github/CopilotForXcode GitHub Copilot for Xcode is an AI coding assistant plugin for Xcode providing code completions, chat, agent mode, and code review for Swift… | 86 | 6338 | active |
| MaiBot MaiBot (MaiSaka) is an open-source, LLM-powered conversational agent designed to behave like a warm, human-like 'digital life form' in QQ g… | 84 | 6338 | active |
| modelscope/FunClip FunClip is an open-source, locally deployed video clipping tool that uses FunASR Paraformer models for speech recognition and subtitle gene… | 93 | 6337 | active |
| PurpleAILAB/Decepticon Decepticon is an autonomous AI red-team hacking agent that uses LLMs (built on LangChain/LangGraph) to plan and execute context-aware offen… | 83 | 6328 | active |
| Harbor Harbor is a Python framework from the creators of Terminal-Bench for evaluating and optimizing AI agents and language models in containeriz… | 86 | 6309 | active |
| Sylinko/Everywhere Everywhere is a cross-platform desktop AI assistant with on-screen awareness that reads the context of whatever app you're using via access… | 80 | 6295 | active |
| NEKOparapa/AiNiee AiNiee is an AI-powered translation tool focused on automatically translating complex long-form content such as RPG/SLG game text, Epub/TXT… | 93 | 6293 | active |
| neuphonic/neutts NeuTTS is a collection of open-source, on-device text-to-speech models built on small LLM backbones, with instant voice cloning from as lit… | 57 | 6286 | active |
| Eigenwise/atomic-agents Atomic Agents is a lightweight, modular Python framework for building agentic AI pipelines and applications from small, composable componen… | 91 | 6284 | active |
| meta-pytorch/gpt-fast A minimal (<1000 lines) PyTorch-native implementation of fast transformer text generation, demonstrating low-latency LLM inference with int… | 41 | 6254 | active |
| meta-llama/llama3 The official Meta repository for Llama 3, providing model weights download scripts, tokenizer, and minimal example code for running inferen… | 10 | 29219 | maintenance |
| vllm-project/semantic-router vLLM Semantic Router is an open-source programmable routing and control layer for building Mixture-of-Models systems across heterogeneous L… | 81 | 6233 | active |
| jihe520/MathModelAgent MathModelAgent is a multi-agent LLM application that automates mathematical modeling competitions end-to-end: it analyzes a problem, builds… | 87 | 6220 | active |
| Q00/ouroboros Ouroboros is an 'Agent OS' that manages AI coding agent workflows through interview-gated requirement capture, staged evaluation, and a bud… | 79 | 6215 | active |
| Shaunwei/RealChar RealChar is an open-source application for creating, customizing, and talking to AI characters/companions in realtime via voice or text. It… | 45 | 6211 | active |
| PrefectHQ/marvin Marvin is a Python framework for building AI applications with LLMs, offering structured-output utilities (extract, cast, classify, generat… | 85 | 6198 | active |
| ByteDance-Seed/Bagel BAGEL is an open-source unified multimodal foundation model with 7B active parameters (14B total) trained on interleaved multimodal data. I… | 52 | 6191 | active |
| microsoft/fara Fara1.5 is a family of open-weight computer use agent (CUA) models from Microsoft Research, released at 4B, 9B, and 27B scales and built on… | 56 | 6186 | active |
| microsoft/TaskWeaver TaskWeaver is a code-first agent framework from Microsoft that plans and executes data analytics tasks by generating and running code snipp… | 10 | 6170 | active |
| YILING0013/AI_NovelGenerator A Python GUI application that uses large language models to generate multi-chapter long-form novels with automatic context continuity, fore… | 62 | 6156 | active |
| wenda-LLM/wenda Wenda is a self-hosted LLM invocation platform designed for efficient content generation in constrained environments, integrating local/off… | 31 | 6148 | active |
| FlashDrive DFlash is a lightweight block diffusion model used as a draft model for speculative decoding of large language models, drafting entire toke… | 65 | 6134 | active |
| Semantic Kernel Semantic Kernel is a model-agnostic SDK and orchestration framework from Microsoft for building AI agents and multi-agent systems in .NET, … | 90 | 28582 | maintenance |
| KServe KServe is a CNCF incubating, Kubernetes-native platform for serving both generative and predictive AI models at scale. It provides a standa… | 94 | 6133 | stable |
| Cactus Cactus is a hybrid edge-cloud AI inference engine for mobile devices, wearables, smart home devices, and robots, built in C++ with custom q… | 84 | 6111 | active |
| google/agents-cli A Python CLI plus agent skills that equip coding assistants (Claude Code, Codex, Antigravity, etc.) to build, evaluate, and deploy AI agent… | 82 | 6094 | active |
| princeton-nlp/tree-of-thought-llm Official implementation of the Tree of Thoughts (ToT) framework for deliberate LLM problem solving, generalizing chain-of-thought prompting… | 21 | 6071 | stable |
| PawanOsman/OpenCursor OpenCursor is an open-source, Cursor-like AI coding agent distributed as a VS Code extension. It provides agentic chat that reads workspace… | 97 | 6023 | active |
| microsoft/Webwright Webwright is a Python framework from Microsoft that turns coding LLMs into state-of-the-art browser agents by giving them a terminal to lau… | 54 | 6011 | active |
| gluonfield/enchanted Enchanted is an open-source iOS/macOS/visionOS chat app that connects to a self-hosted Ollama server to chat with private local language mo… | 69 | 6008 | active |
| TideDra/zotero-arxiv-daily A Python tool deployed as a GitHub Action that recommends new arXiv (and bioRxiv/medRxiv/chemRxiv) papers daily based on the contents of yo… | 79 | 6002 | active |
| amazon-science/chronos-forecasting Chronos is a Python library providing pretrained foundation models for time series forecasting, including Chronos-2 which handles univariat… | 91 | 5999 | active |
| Michael-A-Kuykendall/shimmy Shimmy is a single-binary, OpenAI-compatible LLM inference server written in pure Rust, running GGUF models on a WebGPU-based engine (Airfr… | 86 | 5983 | active |
| 21st-dev/magic-mcp Magic MCP is a legacy compatibility proxy package that forwards MCP requests to the unified 21st MCP server, which lets AI coding agents li… | 73 | 5972 | active |
| kuafuai/DevOpsGPT DevOpsGPT is a multi-agent system that combines LLMs with DevOps tools to convert natural language requirements into working software. It a… | 64 | 5963 | active |
| fla-org/flash-linear-attention A PyTorch library providing hardware-efficient implementations of emerging sequence model architectures, including linear attention, sparse… | 86 | 5940 | active |
| mostlygeek/llama-swap A Go-based proxy service that lets you run multiple local AI inference servers (llama.cpp, vllm, ComfyUI, etc.) and hot-swap between models… | 86 | 5922 | active |
| samchon/typia Typia is a TypeScript transformer library that compiles pure TypeScript types into super-fast runtime validators, JSON serializers/parsers,… | 93 | 5921 | active |
| Giskard-AI/giskard-oss Giskard is an open-source Python library for testing, evaluating, and red-teaming LLM-based agents and RAG systems. Its v3 rewrite provides… | 95 | 5910 | active |
| dograh-hq/dograh Dograh is an open-source, self-hostable voice AI platform for building production voice agents, positioned as an alternative to Vapi and Re… | 85 | 5903 | active |
| SamuelSchmidgall/AgentLaboratory Agent Laboratory is an end-to-end autonomous research workflow framework that uses specialized LLM-driven agents to assist human researcher… | 34 | 5869 | active |
| sqlchat/sqlchat SQL Chat is a chat-based SQL client and editor that lets users query, modify, add, and delete data using natural language, powered by OpenA… | 62 | 5847 | active |
| MadcowD/ell ell is a lightweight Python library for language model programming that treats prompts as functions (language model programs) rather than s… | 32 | 5847 | active |
| OpenAI PHP OpenAI PHP is a community-maintained PHP SDK for interacting with the OpenAI API, covering chat, responses, embeddings, audio, images, fine… | 94 | 5836 | active |
| iyaja/llama-fs LlamaFS is a self-organizing file manager that uses the Llama 3 model to automatically rename and organize files based on their content and… | 37 | 5832 | active |
| HKUDS/AI-Researcher AI-Researcher is an autonomous research system that orchestrates the full scientific research pipeline—from literature review and hypothesi… | 38 | 5815 | active |
| google-deepmind/gemma The official JAX-based Python library from Google DeepMind for running, sampling from, and fine-tuning the Gemma family of open-weight larg… | 85 | 5796 | active |
| mindcraft-bots/mindcraft Mindcraft is a JavaScript application that connects LLM-powered AI agents to Minecraft via the Mineflayer library, letting language models … | 75 | 5776 | active |
| memovai/mimiclaw MimiClaw is a bare-metal AI agent framework written in pure C that runs an OpenClaw-style personal AI assistant on a $5 ESP32-S3 microcontr… | 64 | 5757 | active |
| dnhkng/GLaDOS A real-life implementation of GLaDOS, the sardonic AI from Valve's Portal series, built as a proactive voice assistant with vision, memory,… | 66 | 5735 | active |
| Gemma The official PyTorch implementation of Google's Gemma family of open large language models, including text-only and multimodal variants. It… | 10 | 5710 | active |
| serge-chat/serge Serge is a self-hosted web chat interface for running LLMs locally via llama.cpp, built with a SvelteKit frontend and FastAPI backend. It s… | 10 | 5709 | active |