Open-source llm-inference software
3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
3358 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| tairov/llama2.mojo A single-file implementation of Llama 2 transformer inference written in pure Mojo, leveraging SIMD and vectorization for fast CPU inferenc… | 71 | 2129 | active |
| zilliztech/GPTCache GPTCache is a Python library for building a semantic cache that stores and retrieves LLM responses based on embedding similarity, reducing … | 65 | 8209 | maintenance |
| ozgrozer/ai-renamer A Node.js CLI tool that uses local AI models (via Ollama or LM Studio) or OpenAI to intelligently rename files based on their contents, inc… | 24 | 2123 | active |
| google-gemini/genai-processors GenAI Processors is a lightweight Python library for building modular, asynchronous, and composable AI pipelines around Gemini and other ge… | 75 | 2122 | active |
| LC1332/Chat-Haruhi-Suzumiya An open-source role-playing chatbot project that revives anime characters like Haruhi Suzumiya using large language models, mimicking their… | 30 | 2120 | active |
| ModelEngine-Group/fit-framework FIT is an enterprise-grade AI development framework for the Java ecosystem, combining a polyglot function engine (Java/Python/C++), a strea… | 60 | 2117 | active |
| deepspeedai/DeepSpeed-MII DeepSpeed-MII is a Python library for high-throughput, low-latency large language model inference, built on DeepSpeed. It provides blocked … | 39 | 2114 | active |
| OpenAutoCoder/Agentless Agentless is a Python framework that automatically solves software development problems (bug fixing) using LLMs without an agent loop. It f… | 15 | 2114 | active |
| KEV0143/Telegram-MedicalBot A Telegram-based user support system for medical inquiries that manages user questions through a ticket system. It generates suggested resp… | 40 | 2109 | active |
| itigges22/ATLAS ATLAS is a local-first coding agent that wraps compact open models with planning, candidate generation, quality scoring, sandboxed testing,… | 83 | 2106 | active |
| 4thfever/cultivation-world-simulator An open-source AI-driven Cultivation (Xianxia) world simulator where every cultivator NPC is an independent LLM-powered agent that observes… | 77 | 2106 | active |
| arcsin1/oh-my-ppt Oh My PPT is a local-first, open-source desktop application (Electron + React + TypeScript) that uses AI to generate editable HTML-based pr… | 80 | 2104 | active |
| QwenLM/Qwen2-Audio Official repository for Qwen2-Audio, a 7B-parameter large audio-language model from Alibaba Cloud that accepts audio inputs and responds to… | 28 | 2104 | active |
| kimjammer/Neuro A local recreation of the Neuro-Sama AI VTuber that runs open-source LLMs on consumer hardware, combining realtime speech-to-text, text-to-… | 28 | 2101 | active |
| bigcode-project/starcoder2 StarCoder2 is a family of open code generation language models (3B, 7B, 15B) trained on 600+ programming languages from The Stack v2, with … | 27 | 2095 | active |
| Mangi-11/Eta Eta is a third-party system-level AI agent for Android that uses Root and LSPosed to bypass app sandboxing, giving connected LLMs (BYOK: Ch… | 82 | 2091 | active |
| GordenSun/GordenSuperPPTSkills A set of Codex agent skills that use GPT image generation and vision to create image-format PPT slides and convert them into fully editable… | 49 | 2090 | active |
| vitoplantamura/OnnxStream A lightweight C++ inference library for ONNX models that streams weights to run large models in very little memory, accelerated by XNNPACK.… | 56 | 2089 | active |
| ex3ndr/llama-coder Llama Coder is a self-hosted GitHub Copilot replacement implemented as a VS Code extension. It uses Ollama running CodeLlama or Stable Code… | 27 | 2088 | active |
| BetaStreetOmnis/xhs_ai_publisher A desktop application for AI-powered content creation and automated publishing on Xiaohongshu (Rednote), built with PyQt5, FastAPI, and Pla… | 64 | 2086 | active |
| boldsoftware/meat meat is a Go CLI tool that uses an LLM to abridge a code diff into a 'reading diff', reducing agent-written changes to the conceptually imp… | 53 | 2081 | active |
| stepfun-ai/Step-3.5-Flash Step 3.5 Flash is an open-source 196B-parameter sparse Mixture-of-Experts foundation model (11B active per token) from StepFun, focused on … | 46 | 2080 | active |
| showlab/Code2Video Code2Video is a research framework from Show Lab (NUS) that generates educational videos by having multi-agent LLM pipelines write and rend… | 59 | 2076 | active |
| jd-opensource/OxyGent OxyGent is an open-source Python framework that unifies tools, models, and agents into modular 'Oxy' components for building multi-agent sy… | 57 | 2074 | active |
| handy-computer/transcribe.cpp A C/C++ speech-to-text inference library built on the ggml runtime that runs 16+ ASR model families (Whisper, Parakeet, Canary, Moonshine, … | 82 | 2071 | active |
| OpenMOSS/MOSS-Transcribe-Diarize MOSS-Transcribe-Diarize 0.9B is an open-source end-to-end audio understanding model that jointly performs multi-speaker speech transcriptio… | 57 | 2064 | active |
| ReAgent auto-re-agent is an open-source AI reverse-engineering agent that combines Ghidra binary analysis with LLMs (Claude, Codex, OpenAI-compatib… | 78 | 2063 | active |
| LangGraph4j LangGraph4j is a Java library for building stateful, multi-agent applications with LLMs, inspired by the Python LangGraph project. It lets … | 89 | 2060 | active |
| coder/xum Xum (formerly Mux) is an open-source desktop application for running multiple AI coding agents in parallel, each in its own isolated worksp… | 81 | 2057 | active |
| microsoft/foundry-dev-tools Microsoft Foundry Toolkit (formerly AI Toolkit) is a Visual Studio Code extension for building, testing, and deploying AI agents and models… | 81 | 2056 | active |
| Nutlope/llamatutor LlamaTutor is an open-source AI personal tutor web application powered by Meta's Llama 3.1 70B model via Together AI inference. It is built… | 62 | 2051 | active |
| Yi Yi is a family of open-source large language models trained from scratch by 01.AI, including base and chat models in multiple sizes, with b… | 28 | 7830 | maintenance |
| noamgat/lm-format-enforcer A Python library that enforces output formats like JSON Schema or regex on language models by filtering allowed tokens at each generation s… | 60 | 2040 | active |
| akdeb/ElatoAI ElatoAI is an open-source platform for running realtime voice AI conversations on ESP32/Arduino hardware, supporting 100+ STT, LLM, and TTS… | 63 | 2034 | active |
| Doorman11991/smallcode SmallCode is a terminal-native AI coding agent optimized for small local LLMs (8B-35B parameters) running on consumer hardware. It compensa… | 70 | 2031 | active |
| CaviraOSS/PageLM PageLM is an open-source, AI-powered education platform inspired by NotebookLM that transforms study materials like PDFs and notes into int… | 62 | 2028 | active |
| Tencent-Hunyuan/HunyuanOCR HunyuanOCR-1.5 is a lightweight end-to-end OCR vision-language model from Tencent, with a unified inference environment, llama.cpp PC-side … | 60 | 2025 | active |
| IBM/mcp-cli A Python command-line client for interacting with Model Context Protocol (MCP) servers, integrating LLMs via CHUK Tool Processor and CHUK-L… | 10 | 2024 | active |
| memochou1993/gpt-ai-assistant GPT AI Assistant is an application that connects the OpenAI API with the LINE Messaging API, letting users chat with their own AI assistant… | 58 | 7742 | maintenance |
| Zoo-Code-Org/Zoo-Code Zoo Code is an open-source AI coding assistant delivered as a VS Code extension (with JetBrains/Open VSX support), continuing development o… | 82 | 2020 | active |
| tatsu-lab/alpaca_eval AlpacaEval is an LLM-based automatic evaluation framework for instruction-following language models, producing win rates against a GPT-4 ba… | 33 | 2018 | active |
| 64bit/async-openai async-openai is a Rust library providing typed, async clients for the OpenAI API, covering chat completions, responses, embeddings, assista… | 94 | 2016 | active |
| WuKongOpenSource/Wukong-AICRM Wukong AICRM is an open-source AI-powered CRM positioned as a Salesforce alternative, built in Java. It combines customer management, task … | 63 | 2015 | active |
| deer-flow/llm-space LLM Space is a local-first desktop application for building, tracing, debugging, and evaluating LLM agents. It lets developers prototype ag… | 80 | 2010 | active |
| shcherbak-ai/contextgem ContextGem is an open-source Python LLM framework for extracting structured data and insights from documents with minimal code. It provides… | 90 | 2009 | active |
| FireRedASR2S FireRedASR is a family of open-source industrial-grade automatic speech recognition models supporting Mandarin, Chinese dialects, and Engli… | 49 | 2005 | active |
| gpt4free-ts A TypeScript replication of gpt4free that provides free access to GPT-4 and other LLM models by reverse-engineering various websites offeri… | 21 | 7666 | maintenance |
| mil-tokyo/webdnn WebDNN is a framework for running deep neural network inference directly in the web browser, accepting ONNX models without Python preproces… | 68 | 2003 | active |
| Niek/chatgpt-web A single-page web interface for OpenAI-compatible chat APIs, built with Svelte, where users bring their own API key and chats are stored pr… | 73 | 2001 | active |
| patterns-ai-core/langchainrb Langchain.rb is a Ruby library for building LLM-powered applications, providing a unified interface to many LLM providers (OpenAI, Anthropi… | 73 | 2001 | active |
| chaidiscovery/chai-lab Chai-1 is a state-of-the-art multi-modal foundation model for biomolecular structure prediction, handling proteins, small molecules, DNA, R… | 62 | 2001 | active |
| zai-org/GLM-130B GLM-130B is an open bilingual (English and Chinese) 130-billion-parameter dense language model pre-trained with the General Language Model … | 32 | 7650 | maintenance |
| win4r/AISuperDomain Aila (AISuperDomain) is a cross-platform desktop application that lets users send one prompt to 10+ AI chat services (ChatGPT, Gemini, Clau… | 60 | 1999 | active |
| LingyiChen-AI/JadeAI JadeAI is an open-source, AI-powered resume builder with drag-and-drop editing, 50+ professional templates, and real-time AI optimization i… | 77 | 1998 | active |
| Strix Halo AI Toolboxes A collection of pre-built Toolbx/Docker containers ('toolboxes') for running llama.cpp LLM inference on AMD Ryzen AI Max 'Strix Halo' integ… | 64 | 1998 | active |
| aws/aws-toolkit-vscode AWS Toolkit is a Visual Studio Code extension that connects the IDE to AWS resources, supporting credential management, Lambda debugging vi… | 94 | 1996 | active |
| glebkudr/shotgun_code Shotgun is a desktop application that packages a local codebase into a structured context payload for LLMs, with AI-powered file selection … | 46 | 1995 | active |
| antirez/iris.c Iris is a pure C inference pipeline that generates images from text prompts using open-weights diffusion transformer models like FLUX.2 Kle… | 43 | 1994 | active |
| j4wg/interview-coder-withoupaywall-opensource A free, open-source desktop application (CodeInterviewAssist) that provides AI-powered coding interview assistance, including problem analy… | 34 | 1994 | active |
| openakita/openakita OpenAkita is an open-source multi-agent AI assistant framework in Python that orchestrates multiple cooperating agents with skills, plugins… | 76 | 1990 | active |
| nhaouari/obsidian-textgenerator-plugin An open-source Obsidian plugin that brings generative AI text generation into the note-taking workflow, supporting providers like OpenAI, A… | 86 | 1988 | active |
| NVIDIA/Stable-Diffusion-WebUI-TensorRT An NVIDIA extension for the Stable Diffusion Web UI (Automatic1111) that accelerates image generation using TensorRT-optimized engines on R… | 19 | 1985 | active |
| filiksyos/gitreverse GitReverse is a web application that turns any public GitHub repository into a single synthetic prompt that could have been used to vibe-co… | 60 | 1984 | active |
| run-llama/notebookllama NotebookLlaMa is an open-source, Python-based alternative to Google's NotebookLM, backed by LlamaCloud for document ingestion and retrieval… | 49 | 1976 | active |
| firecrawl/fireplexity Fireplexity is an open-source, Perplexity-style AI search engine built with TypeScript that delivers answers with real-time citations, stre… | 33 | 1975 | active |
| OpenMotionLab/MotionGPT MotionGPT is a unified motion-language model that treats 3D human motion as a foreign language by converting motion into discrete motion to… | 30 | 1974 | active |
| openlm-research/open_llama OpenLLaMA is a permissively licensed (Apache-2.0) open reproduction of Meta AI's LLaMA, releasing 3B, 7B, and 13B pretrained model weights … | 30 | 7527 | maintenance |
| SaiAkhil066/CORTEX-AI-SUPER-RAG CORTEX RAG is a local-first, agentic retrieval-augmented generation application that lets users upload PDFs and ask questions with cited an… | 64 | 1973 | active |
| jd-opensource/JoyAI-VL-Interaction JoyAI-VL-Interaction is an open 8B-scale vision-language interaction model with a complete deployable real-time streaming system, including… | 59 | 1970 | active |
| StarCoder StarCoder is a 15B-parameter code language model trained on 80+ programming languages, and this repository hosts the fine-tuning and infere… | 30 | 7501 | maintenance |
| ROCm/FastFlowLM FastFlowLM (FLM) is an NPU-first LLM inference runtime purpose-built and deeply optimized for AMD Ryzen AI NPUs (XDNA2), offering an Ollama… | 87 | 1967 | active |
| StanfordBDHG/HealthGPT HealthGPT is an experimental open-source iOS app built on the Stanford Spezi framework that lets users query their Apple Health data using … | 77 | 1965 | active |
| siliconflow/onediff OneDiff is an out-of-the-box acceleration library for diffusion models, providing PyTorch compilation tools and optimized GPU kernels. It i… | 44 | 1965 | active |
| showlab/computer_use_ootb An out-of-the-box desktop GUI agent that lets vision-language models like Claude 3.5 Computer Use, ShowUI, and UI-TARS control Windows and … | 29 | 1965 | active |
| mlc-ai/xgrammar XGrammar is an open-source library for efficient, flexible, and portable structured generation in LLM inference via constrained decoding. I… | 91 | 1964 | active |
| Yuliang-Liu/Monkey Monkey is a large multi-modal model (LMM) research project from CVPR 2024 that improves image understanding via higher input resolution and… | 71 | 1956 | active |
| huggingface/chat-macOS HuggingChat macOS is a native Swift chat application for macOS that provides access to open-source AI chat models from Hugging Face. It off… | 10 | 1954 | active |
| Darkatse/TauriTavern TauriTavern is a native rewrite of SillyTavern built with Tauri v2 and a Rust backend, providing the classic LLM chat/roleplay frontend as … | 86 | 1948 | active |
| carlrobertoh/ProxyAI ProxyAI is an open-source AI coding copilot plugin for JetBrains IDEs like IntelliJ and PyCharm. It connects to models from OpenAI, Anthrop… | 75 | 1935 | active |
| wuyoscar/Internal-Safety-Collapse ISC-Bench/TVD is a research framework for studying 'Internal Safety Collapse' in frontier LLMs, where agents placed in adversarial codespac… | 60 | 1935 | active |
| SMNETSTUDIO/WeChat-AI A self-hosted WeChat roleplay chatbot service written in TypeScript that connects directly to Tencent iLink, stores data in remote Redis, a… | 65 | 1931 | active |
| YoMo YoMo is an open-source serverless LLM function calling framework for building fast, geo-distributed AI agents, written in Rust with a QUIC-… | 97 | 1929 | active |
| MaximeRivest/riddle A diary-style application for the reMarkable Paper Pro e-ink tablet that turns handwritten pages into conversations with an LLM. You write … | 69 | 1922 | active |
| LingyiChen-AI/AIComicBuilder AI Comic Builder is a self-hosted Next.js web application that turns scripts into fully animated comic videos through an automated AI pipel… | 61 | 1921 | active |
| IPADS-SAI/MobiAgent MobiAgent is a systematic framework for building customizable GUI agents that operate mobile phones, comprising the MobiMind agent model fa… | 54 | 1920 | active |
| Robbyant/lingbot-va LingBot-VA is an autoregressive diffusion framework that unifies video world modeling and robot policy learning in a single interleaved vid… | 53 | 1918 | active |
| alexpinel/Dot Dot is a standalone Electron desktop application for chatting with your documents using fully local LLMs and Retrieval Augmented Generation… | 16 | 1915 | active |
| Dicklesworthstone/pi_agent_rust pi_agent_rust is a native, single-binary AI coding agent CLI written in Rust with zero unsafe code, ported from Mario Zechner's Pi Agent. I… | 83 | 1914 | active |
| deepseek-ai/DeepSeek-LLM DeepSeek LLM is a family of open-source large language models (7B and 67B, Base and Chat variants) trained from scratch on 2 trillion token… | 27 | 7251 | maintenance |
| AiPPT AiPPT (文多多) is a commercial-grade AI PowerPoint generation project that creates presentations from topics, files, or URLs, and can parse PP… | 46 | 1910 | active |
| Ollamac Ollamac is a native macOS application that provides a chat interface for interacting with local Ollama models. It supports all Ollama model… | 30 | 1910 | active |
| melih-unsal/DemoGPT DemoGPT is a Python framework and autonomous agent that generates LangChain-based LLM agent applications from natural-language prompts, bun… | 51 | 1906 | active |
| AutoFigure AutoFigure-Edit is a Python application that converts scientific paper method sections into fully editable SVG figures using large language… | 53 | 1905 | active |
| ShenSeanChen/waku-agent Waku Agent is a local-first AI agent harness in Python that exposes the full agent stack — harness, loop, memory, and eval/LLM-Ops — as sma… | 79 | 1903 | active |
| vllm-project/vllm-metal A community-maintained hardware plugin that enables vLLM to run LLM inference on Apple Silicon Macs using MLX as the primary compute backen… | 80 | 1898 | active |
| ivanfioravanti/chatbot-ollama An open-source web-based chat UI for Ollama, forked from chatbot-ui, that lets users chat with locally hosted LLMs. It supports document (P… | 44 | 1895 | active |
| bhaskatripathi/pdfGPT pdfGPT is an open-source Python application that lets users chat with the contents of PDF files using GPT capabilities. It implements a sim… | 59 | 7162 | maintenance |
| zakirkun/guardian-cli Guardian is a Python CLI tool that automates penetration testing workflows using LLM providers (OpenAI, Claude, Gemini, Ollama, and others)… | 63 | 1889 | active |
| SqueezeAILab/LLMCompiler LLMCompiler is a Python framework from an ICML 2024 paper that orchestrates parallel function calling with LLMs. It uses a planner, task fe… | 27 | 1889 | active |
| NVIDIA-AI-IOT/Lidar_AI_Solution NVIDIA's collection of GPU-accelerated Lidar AI inference solutions for autonomous driving, including optimized implementations of PointPil… | 69 | 1885 | active |