Open-source llm-inference software
3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
3358 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| lastmile-ai/aiconfig AIConfig is an open-source framework that manages generative AI prompts, models, and model parameters as JSON-serializable configs, separat… | 45 | 1088 | active |
| rhymes-ai/Aria Aria is an open multimodal native Mixture-of-Experts (MoE) model with 25.3B total parameters (3.9B activated per token) and a 64K multimoda… | 23 | 1088 | active |
| XInTheDark/raycast-g4f A Raycast extension that provides free access to GPT-4, Claude, Llama, Gemini and other AI models via GPT4Free providers, with full support… | 51 | 1087 | active |
| ufy2024/AuC AuC (Agents-ufy-Core) is a Python single-agent framework built on asyncio with a pluggable ReAct reasoning loop, LLM adapters, tiered tool … | 53 | 1086 | active |
| tigicion/dao-code Dao Code is an open-source terminal-native AI coding agent written in TypeScript, targeting DeepSeek V4 with 1M context and Claude Code-com… | 75 | 1084 | active |
| QwenLM/Qwen2.5-Math Qwen2.5-Math is a series of math-specialized large language models (1.5B/7B/72B base and instruct variants plus a 72B reward model) built o… | 24 | 1084 | active |
| mit-han-lab/streaming-vlm StreamingVLM is a vision-language model framework from MIT Han Lab for real-time understanding of effectively infinite video streams. It ma… | 35 | 1083 | active |
| FalkorDB/QueryWeaver QueryWeaver is an open-source Text2SQL tool that converts plain-English questions into SQL queries using graph-powered schema understanding… | 82 | 1082 | active |
| HannesStark/boltzgen BoltzGen is an open-source all-atom generative diffusion model for designing protein and peptide binders against arbitrary biomolecular tar… | 75 | 1081 | active |
| GradientHQ/symphony-coord Symphony-Coord is a decentralized multi-agent framework that treats agent selection as an online multi-armed bandit problem, allowing roles… | 60 | 1081 | active |
| LangSmith Official Python and JavaScript/TypeScript client SDKs for the LangSmith platform, enabling tracing, debugging, evaluation, and monitoring o… | 10 | 1081 | active |
| ikechan8370/chatgpt-plugin A ChatGPT/LLM chat plugin for Yunzai-Bot (Miao-Yunzai) QQ bots, built on the Chaite kernel. It provides multi-model channels, presets, grou… | 74 | 1080 | active |
| BruceMacD/chatd Chatd is an Electron desktop application that lets you chat with your local documents using a local large language model (Mistral-7B) via O… | 18 | 1080 | active |
| FuJacob/cotabby Cotabby is an open-source macOS menu bar application that provides local, on-device AI autocomplete (ghost text) in nearly any text field a… | 77 | 1078 | active |
| Pokee-AI/PokeeResearchOSS An open-source repository for Pokee's 7B-parameter DeepResearch agent that performs multi-turn web search and content reading to answer com… | 35 | 1078 | active |
| FINN FINN is an open-source dataflow compiler from AMD/Xilinx that generates highly efficient FPGA accelerators for quantized neural network (QN… | 85 | 1077 | active |
| tryAGI/LangChain LangChain .NET is a C#/.NET implementation of the LangChain framework for building LLM-powered applications through composability. It provi… | 64 | 1077 | active |
| vstorm-co/pydantic-deepagents Pydantic Deep Agents is an open-source, self-hosted terminal AI assistant (a Claude Code alternative) plus the Python framework behind it, … | 79 | 1076 | active |
| mskayyali/nodepad nodepad is a spatial note-taking web application where notes are placed on a canvas and AI quietly classifies them, infers connections, and… | 54 | 1074 | active |
| sipeed/TinyMaix TinyMaix is a tiny neural network inference library for microcontrollers (TinyML), with core code under 400 lines and a .text section under… | 32 | 1074 | active |
| dynamiq-ai/dynamiq Dynamiq is an open-source Python orchestration framework for building agentic AI and LLM applications, covering agents, RAG pipelines, work… | 85 | 1071 | active |
| Snow Snow CLI is a terminal-based AI coding agent written in TypeScript that can read, write, and refactor code directly from the command line. … | 82 | 1069 | active |
| codeacme17/examor Examor is a self-hosted web application that generates exams and review questions from your knowledge notes using LLMs like GPT-4 and Claud… | 28 | 1068 | active |
| zai-org/GLM-TTS GLM-TTS is a Python-based text-to-speech synthesis system built on large language models, using a two-stage LLM plus Flow model architectur… | 47 | 1066 | active |
| Dataherald/dataherald Dataherald is an open-source natural language-to-SQL engine for enterprise question answering over relational databases, exposing an API th… | 20 | 3648 | maintenance |
| CRui5in/paper-ppt-agent A self-hosted web application that uses a multi-agent AI pipeline (Strategist, Executor, Critic) to convert academic papers from PDF or LaT… | 73 | 1065 | active |
| sybil-solutions/codex-shim A local Python/aiohttp server that exposes an OpenAI Responses-compatible endpoint on loopback, letting Codex Desktop use any BYOK model (O… | 55 | 1065 | active |
| Felix3322/PotPlayer_ChatGPT_Translate A PotPlayer plugin that integrates OpenAI-compatible AI APIs (including ChatGPT and Ollama) to translate video subtitles in real time. It u… | 88 | 1064 | active |
| metaskills/experts Experts.js is a JavaScript library that simplifies creating and deploying OpenAI Assistants via the Assistants API, hiding Run object manag… | 25 | 1064 | active |
| pepperoni21/ollama-rs A Rust client library for interacting with the Ollama API, supporting completion generation, streaming, chat, model management, embeddings,… | 84 | 1063 | active |
| PatterAI/Patter Patter is an open-source, MIT-licensed SDK (Python and TypeScript) that connects AI agents to real phone calls, handling telephony, speech-… | 73 | 1062 | active |
| bigcode-project/bigcode-evaluation-harness A framework for evaluating autoregressive code generation language models on benchmarks like HumanEval, MBPP, MultiPL-E, and HumanEvalPack.… | 34 | 1062 | active |
| centerforaisafety/HarmBench HarmBench is a standardized, open-source evaluation framework for automated red teaming of large language models, comparing attack methods … | 26 | 1062 | active |
| mybigday/llama.rn A React Native binding of llama.cpp that enables on-device LLM inference on iOS and Android. It supports GPU/NPU acceleration (Metal, Hexag… | 93 | 1061 | active |
| onnx/onnx-mlir ONNX-MLIR is a compiler infrastructure built on LLVM/MLIR that lowers ONNX neural network graphs into executable code, MLIR files, LLVM byt… | 86 | 1061 | active |
| Tencent/WeSmartFlow WeSmartFlow is an agent-native adaptive learning framework that models the full learning process with ReAct-based tutoring agents, graph me… | 57 | 1059 | active |
| InternRobotics/PointLLM PointLLM is a multimodal large language model that understands colored 3D point clouds of objects, built on a point cloud encoder fused wit… | 62 | 1058 | active |
| pytorch/torchchat torchchat is a compact PyTorch-native codebase for running large language models locally via Python, a C/C++ runner on desktop/server, or o… | 10 | 3611 | maintenance |
| metauto-ai/GPTSwarm GPTSwarm is a Python library for building LLM-based agents as computational graphs, with modules for agent graphs, memory, LLM backends, an… | 52 | 1055 | active |
| Minima-AI-Inc/minima Minima is an open-source, on-premises conversational RAG system deployed via Docker containers. It supports fully local operation with Olla… | 38 | 1055 | active |
| BAAI-DCAI/Bunny Bunny is a family of lightweight multimodal vision-language models that combine plug-and-play vision encoders (EVA-CLIP, SigLIP) with langu… | 26 | 1054 | active |
| braintrustdata/autoevals Autoevals is a Python and TypeScript library for automatically evaluating AI model outputs using LLM-as-a-judge, heuristic, and statistical… | 91 | 1053 | active |
| Dicklesworthstone/swiss_army_llama A FastAPI-based REST service that exposes local LLM capabilities including text embeddings, completions, semantic similarity, and semantic … | 29 | 1053 | active |
| shyamsaktawat/OpenAlpha_Evolve OpenAlpha_Evolve is an open-source Python framework inspired by DeepMind's AlphaEvolve that uses LLMs to iteratively generate, test, and ev… | 27 | 1053 | active |
| scrya-com/rotorquant RotorQuant is a KV cache compression method for LLM inference that replaces full d×d orthogonal rotations with block-diagonal Clifford roto… | 47 | 1049 | active |
| xuzhougeng/wisp-science Wisp Science is an open-source, local-first desktop AI research workbench that combines an LLM agent with persistent Python/R kernels, MCP-… | 77 | 1047 | active |
| tonbistudio/turboquant-pytorch A from-scratch PyTorch implementation of Google's TurboQuant (ICLR 2026) algorithm for compressing LLM key-value caches, including an impro… | 47 | 1047 | active |
| wrtnlabs/agentica Agentica is a TypeScript agentic AI framework specialized in LLM function calling, enhanced by compiler-driven validation via typia. Agents… | 65 | 1046 | active |
| andyzoujm/representation-engineering Official library for Representation Engineering (RepE), a top-down approach to AI transparency that monitors and manipulates population-lev… | 19 | 1046 | active |
| octos-org/octos Octos is a self-hosted AI agent runtime that runs a personal AI assistant on your own machine, connecting to providers like Anthropic, Open… | 78 | 1044 | active |
| villesau/ai-codereviewer A GitHub Action that uses OpenAI's GPT-4 API to automatically review pull requests and post intelligent feedback comments. It retrieves PR … | 21 | 1044 | active |
| yoheinakajima/instagraph InstaGraph is a Flask web application that converts text input or URLs into knowledge graphs using OpenAI's GPT-3.5. It renders color-coded… | 28 | 3555 | maintenance |
| NVIDIA-NeMo/Skills Nemo-Skills is a collection of Python pipelines for improving the skills of large language models, covering synthetic data generation, mode… | 69 | 1042 | active |
| vercel-labs/tersa Tersa is an open-source visual AI playground built as a Next.js web application where users drag, drop, and connect nodes on a canvas to bu… | 58 | 1042 | active |
| Refact Refact is an open-source, self-hosted AI coding agent written in Rust that plans and executes engineering tasks end-to-end from your IDE, w… | 10 | 3541 | maintenance |
| Nutlope/blinkshot BlinkShot is an open-source web application that generates AI images in real time as you type, powered by the Flux Schnell model via Togeth… | 64 | 1039 | active |
| canwhite/Krebs Krebs is a self-hosted AI gateway with a general-purpose agent focused on attention management. It provides multi-turn dialogue with tool e… | 57 | 1039 | active |
| ricklamers/gpt-code-ui An open-source, self-hosted implementation of OpenAI's ChatGPT Code Interpreter. Users chat with GPT-3.5/GPT-4 models that generate and exe… | 21 | 3536 | maintenance |
| Goekdeniz-Guelmez/Local-NotebookLM A local, open-source alternative to Google's NotebookLM that converts PDF documents into audio content like podcasts, summaries, and interv… | 68 | 1037 | active |
| kuleshov-group/bd3lms BD3-LMs is a research implementation of Block Discrete Denoising Diffusion Language Models that interpolate between autoregressive and diff… | 31 | 1037 | active |
| darrencxl0301/StageRAG StageRAG is a Python framework/blueprint for building production-ready RAG applications with switchable 3-step (Speed) and 4-step (Precisio… | 35 | 1036 | active |
| m-damien/VisualStoryWriting A web application that automatically visualizes a story's chronology, characters, and their movements, letting writers edit the story by ma… | 29 | 1036 | active |
| walter-grace/mac-code A free, local coding agent for Apple Silicon Macs that runs large quantized LLMs (e.g., Qwen 35B) via llama.cpp, using techniques like flas… | 46 | 1035 | active |
| wxywb/history_rag A Chinese history question-answering application built with RAG (retrieval-augmented generation) over a corpus of Chinese historical texts,… | 27 | 1035 | active |
| adrida/tracer TRACER is a Python library that learns a routing policy from LLM classification traces, fitting a lightweight traditional ML surrogate to h… | 75 | 1033 | active |
| samuel-vitorino/lm.rs A minimal Rust implementation for running LLM inference locally on CPU without ML libraries, supporting Gemma 2, Llama 3.2, and Phi 3.5 (in… | 24 | 1033 | active |
| kaito-project/kaito KAITO is a Kubernetes operator suite that automates LLM inference, fine-tuning, and RAG engine deployment using simplified CRD APIs. It aut… | 93 | 1028 | active |
| microsoft/Tutel Tutel is Microsoft's optimized Mixture-of-Experts (MoE) library for efficient training and inference of large language models, featuring dy… | 79 | 1028 | active |
| Sunone Aimbot An AI-powered aimbot for first-person shooter games that uses YOLO object detection models (YOLOv8/v10/v12) with TensorRT/ONNX acceleration… | 62 | 1027 | active |
| minimaxir/simpleaichat A Python library providing a minimal, low-complexity interface for building applications on top of chat LLM APIs like ChatGPT and GPT-4. It… | 21 | 3491 | maintenance |
| kuprel/min-dalle min(DALL·E) is a fast, minimal PyTorch port of DALL·E Mini/Mega stripped down for text-to-image inference, with only numpy, requests, pillo… | 27 | 3490 | maintenance |
| hrithikkoduri/WebRover WebRover is an autonomous AI web agent that interprets user input, navigates websites via browser automation, and performs tasks or deep re… | 12 | 1026 | active |
| MeshAnything MeshAnything is an autoregressive transformer model that generates artist-created 3D meshes (up to 1600 faces in V2) aligned with a given s… | 28 | 1023 | active |
| jucasoliveira/terminalGPT TerminalGPT is a Node.js CLI tool that brings ChatGPT-like LLM chat conversations directly into your terminal. It supports multiple LLM pro… | 22 | 1022 | active |
| addyosmani/bg-remove A React + Vite web application that removes image backgrounds entirely in the browser using Transformers.js with the RMBG-1.4 model (and op… | 22 | 1021 | active |
| HumeAI/tada TADA is an open-source speech-language model from Hume AI that generates expressive, high-fidelity speech via text-acoustic dual alignment,… | 48 | 1020 | active |
| XGenerationLab/XiYan-SQL XiYan-SQL is a multi-generator ensemble framework for converting natural language questions into SQL queries, achieving SOTA results on ben… | 56 | 1019 | active |
| deepseek-ai/DeepSeek-Math DeepSeekMath is a 7B open language model specialized in mathematical reasoning, initialized from DeepSeek-Coder and trained on 500B math-re… | 26 | 3457 | maintenance |
| hinterdupfinger/obsidian-ollama An Obsidian plugin that integrates locally running Ollama models into your notes. It provides preconfigured prompts (summarize, explain, re… | 19 | 1018 | active |
| Alpha-VLLM/Lumina-Image-2.0 Lumina-Image 2.0 is an open-source 2.6B-parameter text-to-image generation framework built on a unified Next-DiT architecture with a unifie… | 55 | 1017 | active |
| alumnium-hq/alumnium Alumnium is an AI-native library and MCP server for end-to-end testing that layers natural-language actions, checks, and data extraction on… | 90 | 1016 | active |
| srcbookdev/srcbook Srcbook is a local-first TypeScript notebook application that runs as a CLI with a web interface, supporting executable TypeScript cells, m… | 64 | 3444 | maintenance |
| synthetic-lab/octofriend OctoFriend is a zero-telemetry, terminal-based AI coding assistant that works with any OpenAI- or Anthropic-compatible LLM API, including l… | 65 | 1015 | active |
| talkie-lm/talkie talkie is a Python library and CLI for running inference with the talkie 13B language model family, including a 13B model trained on pre-19… | 49 | 1015 | active |
| voquill/voquill Voquill is an open-source, cross-platform AI voice dictation app that lets users dictate into any desktop application, with AI-powered tran… | 78 | 1012 | active |
| 274056675/springboot-openai-chatgpt A full-stack AI chatbot application built on Spring Boot/Spring Cloud that integrates GPT-3.5, GPT-4, Baidu ERNIE Bot, Stable Diffusion, an… | 33 | 1012 | active |
| ZTE-AICloud/Co-Sight Co-Sight is an open-source Python framework from ZTE for building Manus-like autonomous AI agent systems that generate high-quality researc… | 41 | 1011 | active |
| stitionai/devika Devika is an open‑source autonomous AI software engineer that can interpret high‑level instructions, research, plan, and write code across … | 41 | 19559 | experimental |
| QwenAudio/Fun-Audio-Chat Fun-Audio-Chat is a Large Audio Language Model (8B) for natural, low-latency voice interactions, using dual-resolution speech representatio… | 60 | 1009 | active |
| Fadi002/de4py An open-source, AI-powered toolkit for reverse engineering Python code, combining an LLM-driven deobfuscation engine (via Ollama) with lega… | 88 | 1007 | active |
| xqdoo00o/chatgpt-web A pure JavaScript ChatGPT web client that ships as a single HTML file and talks directly to the OpenAI API (plus DeepSeek, Gemini, and Clau… | 31 | 1007 | active |
| ZJUI-AI4H/Hulu-Med Hulu-Med is a family of open-source transparent generalist medical vision-language models ranging from 4B to 235B parameters, covering text… | 59 | 1006 | active |
| minimaxir/gpt-2-simple A Python package that simplifies fine-tuning OpenAI's GPT-2 text-generation model (124M/355M) on custom text and generating text from the r… | 23 | 3405 | maintenance |
| MoonshotAI/checkpoint-engine Checkpoint-engine is a lightweight Python middleware for updating model weights in-place across LLM inference engines, a critical step in r… | 78 | 1005 | active |
| ViggoZ/producthunt-daily-hot A Python automation tool that generates a daily Chinese-language digest of Product Hunt's top 30 products, using the Product Hunt API and O… | 69 | 1005 | active |
| disler/always-on-ai-assistant A Python reference pattern for an always-on voice AI assistant that listens via RealtimeSTT, thinks with Deepseek-V3 or local Ollama models… | 21 | 1003 | active |
| MichaelYuhe/ai-group-tabs A Chrome extension that uses AI (OpenAI API) to automatically organize and group browser tabs into categories. Users can customize categori… | 18 | 1000 | active |
| WangRongsheng/CareGPT CareGPT is a medical large language model project that aggregates dozens of publicly available medical fine-tuning datasets and open medica… | 10 | 999 | active |
| JIA-Lab-research/MGM Official PyTorch implementation of Mini-Gemini, a multimodal vision-language model framework built on LLaVA that supports dense and MoE LLM… | 26 | 3322 | maintenance |
| XYZ-AI-Lab/axrl AxisRL is an agentic reinforcement learning post-training framework for large language models, built on SGLang for high-throughput rollout … | 53 | 984 | active |