Open-source llm-inference software
3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
3358 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| vLLM vLLM-Omni is a Python framework extending vLLM for efficient inference and serving of omni-modality models, including diffusion transformer… | 84 | 7309 | active |
| gaozhangmin/boxplayer BoxPlayer is a cross-platform desktop application that unifies multiple cloud drives (Aliyun Drive, Baidu, 115, Quark, OneDrive, etc.), loc… | 98 | 7276 | active |
| Zipstack/unstract Unstract is an open-source, LLM-driven platform that extracts structured JSON data from unstructured documents such as PDFs, images, and sc… | 87 | 7270 | active |
| Zyphra/Zonos Zonos-v0.1 is an open-weight text-to-speech model trained on over 200k hours of multilingual speech, with a Python library for inference. I… | 22 | 7236 | active |
| AI Scientist An end-to-end autonomous AI research system that generates hypotheses, runs machine learning experiments, analyzes data, and writes complet… | 42 | 7222 | active |
| ddean2009/MoneyPrinterPlus A Python desktop application that uses AI LLMs to batch-generate short videos with one click, including automatic video mashup/remixing and… | 16 | 7182 | active |
| lintsinghua/DeepAudit DeepAudit is an open-source multi-agent AI system for automated code vulnerability discovery and security auditing, with a React frontend a… | 71 | 7178 | active |
| enricoros/big-AGI Big-AGI is an open-source AI workspace web application that lets users chat with many state-of-the-art LLM providers (OpenAI, Anthropic, Ge… | 91 | 7151 | active |
| htdt/godogen A generator that publishes thin game repos where AI coding agents (Claude Code or Codex) autonomously build complete games for Godot 4, Bev… | 60 | 7137 | active |
| wzpan/wukong-robot wukong-robot is a modular Chinese-language voice assistant / smart speaker project in Python that combines offline wake-word detection, ASR… | 32 | 7127 | active |
| Vespa Vespa is an open-source, distributed AI search platform and serving engine that combines full-text search, vector/tensor search, and machin… | 93 | 7117 | stable |
| katanemo/plano Plano is an AI-native proxy server and data plane for agentic applications, built on Envoy by its core contributors. It centralizes agent r… | 85 | 7092 | active |
| deeppavlov/DeepPavlov DeepPavlov is an open-source Python NLP library built on PyTorch and Hugging Face transformers for developing, training, and deploying stat… | 35 | 6995 | active |
| openai/openai-realtime-agents A Next.js TypeScript demo application showcasing advanced agentic patterns (Chat-Supervisor and Sequential Handoff) built on the OpenAI Rea… | 45 | 6989 | active |
| firebase/firebase-ios-sdk The official Firebase SDK for Apple platforms (iOS, macOS, tvOS, watchOS, visionOS), providing open-source client libraries for Firebase pr… | 99 | 6965 | stable |
| InternLM/MindSearch MindSearch is an open-source, LLM-based multi-agent framework that mimics human cognition to perform deep web searches, similar to Perplexi… | 24 | 6943 | active |
| yihong0618/xiaogpt A Python application that connects Xiaomi AI Speakers to ChatGPT and other LLMs, letting users ask questions starting with a trigger phrase… | 58 | 6921 | active |
| Tencent/AI-Infra-Guard Tencent's full-stack AI red teaming platform that scans AI infrastructure, agents, MCP servers, and skills for vulnerabilities and evaluate… | 85 | 6916 | active |
| TencentQQGYLab/AppAgent AppAgent is an LLM-based multimodal agent framework that operates smartphone apps through human-like interactions such as tapping and swipi… | 28 | 6896 | active |
| dramaclaw/dramaclaw DramaClaw is a self-hosted, general-purpose AIGC video engine that turns scripts into finished films through a single pipeline covering sto… | 81 | 6851 | active |
| Dooy/chatgpt-web-midjourney-proxy A unified web/desktop UI for ChatGPT plus AI image, music, and video generation services like Midjourney, Suno, Luma, Runway, and Flux. It … | 93 | 6795 | active |
| steipete/summarize Summarize is a Node.js CLI and Chrome/Firefox extension that extracts clean text from web pages, PDFs, YouTube videos, podcasts, and audio/… | 84 | 6729 | active |
| langchain-ai/opengpts OpenGPTs is an open-source platform for building and hosting custom AI assistants similar to OpenAI's GPTs and Assistants API. It is built … | 10 | 6707 | active |
| 11cafe/jaaz Jaaz is an open-source multimodal creative assistant application that serves as a privacy-focused, locally usable alternative to Canva and … | 49 | 6689 | active |
| Yuliang-Liu/MonkeyOCR MonkeyOCR is a lightweight large multimodal model (LMM) for document parsing that uses a Structure-Recognition-Relation triplet paradigm to… | 58 | 6650 | active |
| Chanzhaoyu/chatgpt-web A self-hosted ChatGPT demo web application built with Express (Node.js backend) and Vue 3 frontend. It supports two access methods: the off… | 10 | 31414 | maintenance |
| ultraworkers/claw-code-parity A temporary Rust workspace tracking parity work for the claw-code coding agent while the original repo undergoes migration to Rust. It prov… | 10 | 6624 | active |
| elder-plinius/T3MP3ST T3MP3ST is a multi-agent offensive-security framework that turns existing AI coding agents (Claude Code, Codex, Ollama, etc.) into autonomo… | 65 | 6496 | active |
| CodePilot CodePilot is a multi-model AI agent desktop client built with Electron and Next.js that connects to 17+ AI providers through a single inter… | 76 | 6471 | active |
| oomol-lab/pdf-craft pdf-craft is a Python library that converts PDF files into Markdown or EPUB, with a focus on scanned books and documents. It uses OCR (loca… | 89 | 6413 | active |
| WhisperKit A Swift SDK providing turn-key on-device speech AI frameworks for Apple Silicon, including WhisperKit (speech-to-text with Whisper), Speake… | 88 | 6397 | active |
| mayocream/koharu Koharu is a local-first desktop application that automates manga translation using machine learning, combining text/bubble detection, OCR, … | 82 | 6394 | active |
| ThinkInAIXYZ/deepchat DeepChat is an open-source, local-first AI agent desktop client built with Electron that connects to multiple LLM providers like OpenAI, Cl… | 88 | 6369 | active |
| joeseesun/qiaomu-anything-to-notebooklm A Claude Code Skill that ingests content from 15+ sources (WeChat articles, web pages, YouTube, PDFs, EPUB, Office docs, audio) and uploads… | 69 | 6347 | active |
| anthropics/claude-code-security-review A GitHub Action that uses Anthropic's Claude to perform AI-powered security reviews of pull requests, analyzing code diffs for vulnerabilit… | 45 | 6319 | active |
| nickscamara/open-deep-research An open-source clone of OpenAI's Deep Research, built as a Next.js web application. It uses Firecrawl's search and extract APIs to gather l… | 17 | 6291 | active |
| TableProApp/TablePro TablePro is a free, open-source, native macOS (and iOS/iPadOS) database client supporting 29 SQL and NoSQL engines through native drivers, … | 80 | 6286 | active |
| PaddlePaddle/PaddleX PaddleX is a low-code, all-in-one AI development tool built on the PaddlePaddle framework, bundling 200+ pretrained models into 33 producti… | 86 | 6268 | active |
| JimmyLv/BibiGPT-v1 BibiGPT is an AI-powered web application that summarizes videos and audio from platforms like Bilibili, YouTube, TikTok, and podcasts, gene… | 54 | 6241 | active |
| Helicone Helicone is an open-source LLM observability platform and AI gateway that logs, monitors, and analyzes LLM API requests with a one-line int… | 78 | 6224 | active |
| shy3130/tick-stock-panel A self-hosted, zero-ops quantitative workbench for China A-share stock screening, monitoring, and backtesting, built with Python (FastAPI, … | 82 | 6191 | active |
| OpenPencil OpenPencil is an open-source, AI-native vector design application where users generate and edit UI designs on an infinite canvas via natura… | 68 | 6076 | active |
| netease-youdao/LobsterAI LobsterAI is an open-source, desktop-grade AI agent application by NetEase Youdao that performs real work like data analysis, slides, docum… | 76 | 6064 | active |
| truefoundry/trueforge TrueForge is an open-source agent harness that provides the runtime layer turning an LLM into a working agent, with MCP tools, skills, sand… | 79 | 6032 | active |
| ModelEngine-Group/nexent Nexent is a zero-code, self-hosted platform for auto-generating production-grade AI agents from natural language prompts, built on Harness … | 87 | 6012 | active |
| Lemonade Lemonade is a local AI server that runs optimized LLMs (plus image, speech, and embedding models) on your own GPU and NPU, exposing OpenAI-… | 83 | 5974 | active |
| ChaoningZhang/MobileSAM MobileSAM is the official implementation of a lightweight version of Meta's Segment Anything Model (SAM), replacing the heavyweight image e… | 62 | 5885 | stable |
| GPUStack GPUStack is an open-source GPU cluster manager for AI model serving that orchestrates inference engines like vLLM, SGLang, and TensorRT-LLM… | 88 | 5870 | active |
| areal-project/AReaL AReaL is a large-scale asynchronous reinforcement learning system that bridges foundation model training with agent-based applications, sup… | 86 | 5842 | active |
| vercel-labs/open-agents Open Agents is an open-source reference application and template for building cloud-hosted background coding agents on Vercel. It combines … | 57 | 5833 | active |
| AidLux (AidLearning) AidLux (originally AidLearning) is an AIoT development platform that runs a native Ubuntu Linux environment with GUI, deep learning tooling… | 63 | 5812 | active |
| Eventual-Inc/Daft Daft is a high-performance distributed data engine with a Python dataframe API, implemented in Rust, designed for AI and multimodal workloa… | 95 | 5804 | active |
| PySpur-Dev/pyspur PySpur is an open-source visual playground and builder for AI agent workflows, letting engineers build agents via drag-and-drop UI or Pytho… | 61 | 5792 | active |
| coze-dev/coze-loop Coze Loop is an open-source, self-hostable platform for full-lifecycle AI agent development, offering prompt engineering with a visual play… | 72 | 5764 | active |
| RuoYi-AI RuoYi AI is an open-source, enterprise-grade full-stack AI assistant platform built in Java on the RuoYi ecosystem and LangChain4j. It unif… | 91 | 5761 | active |
| OpenBMB/UltraRAG UltraRAG is a low-code Python framework from OpenBMB for building complex RAG pipelines using MCP servers and YAML-based orchestration with… | 86 | 5706 | active |
| antiwork/shortest Shortest is an AI-powered end-to-end testing framework that lets developers write browser tests in plain English, executed via Anthropic's … | 64 | 5684 | active |
| modstart-lib/aigcpanel AIGCPanel is an open-source, all-in-one AI digital human desktop application built with TypeScript, Vue3, and Electron for Windows, macOS, … | 90 | 5680 | active |
| the-open-agent/openagent OpenAgent is a self-hosted, single-binary personal AI assistant platform built in Go that combines LLM chat, RAG knowledge bases, and auton… | 93 | 5670 | active |
| ZenML ZenML is an open-source MLOps framework for building, deploying, and managing production ML pipelines, LLM workflows, and AI agents on any … | 97 | 5601 | active |
| ArcReel/ArcReel ArcReel is an open-source, self-hosted AI video production workspace that turns novels, scripts, or product material into characters, scene… | 79 | 5598 | active |
| remsky/Kokoro-FastAPI A Dockerized FastAPI wrapper around the Kokoro-82M text-to-speech model exposing an OpenAI-compatible speech endpoint with CPU, NVIDIA, AMD… | 87 | 5543 | active |
| HKUDS/ClawTeam ClawTeam is a framework-agnostic Python CLI that orchestrates swarms of AI coding agents (Claude Code, Codex, Cursor, and any CLI agent) to… | 60 | 5536 | active |
| BIT-DataLab/Edit-Banana Edit Banana is an open-source Python framework that converts static images and PDFs of diagrams, flowcharts, and charts into fully editable… | 59 | 5498 | active |
| agentgateway/agentgateway Agentgateway is an open-source, Rust-based proxy and gateway for agentic AI traffic, handling LLM provider routing, MCP tool servers, and A… | 83 | 5420 | active |
| milind-soni/OpenMausBot OpenMausBot is an open-source, local-first desktop chat application (Electron, TypeScript) where each contact is an AI agent powered by Cla… | 77 | 5374 | active |
| nanbingxyz/5ire 5ire is a cross-platform desktop AI assistant and MCP (Model Context Protocol) client that connects to major LLM providers like OpenAI, Ant… | 80 | 5373 | active |
| superduper-io/superduper Superduper is an open-source Python framework for building database-integrated AI applications and agents, connecting models, embeddings, a… | 44 | 5329 | active |
| FunnyWolf/Viper VIPER is a self-hosted red teaming and adversary simulation platform with a web UI, 100+ post-exploitation modules covering MITRE ATT&CK, a… | 78 | 5327 | active |
| beclab/Olares Olares is an open-source personal cloud operating system built on Kubernetes that turns your own hardware into a self-hosted AI platform fo… | 90 | 5299 | active |
| TagSpaces TagSpaces is an open-source, offline-first desktop and mobile application for organizing, tagging, and previewing local files, notes, and t… | 94 | 5297 | active |
| Awarexone/Agentic-Bug-Hunter An AI-powered bug bounty hunting toolkit that automates reconnaissance, vulnerability testing, finding validation, and report generation fo… | 75 | 5294 | active |
| iflytek/astron-rpa AstronRPA is an open-source enterprise-grade Robotic Process Automation desktop application with a visual low-code/no-code designer for aut… | 69 | 5276 | active |
| vitali87/code-graph-rag Code-Graph-RAG is a Python CLI tool and MCP server that parses multi-language codebases with Tree-sitter, builds a knowledge graph of code … | 86 | 5258 | active |
| wenet-e2e/wenet WeNet is a production-first, end-to-end automatic speech recognition (ASR) toolkit built on PyTorch with transformer/conformer models. It p… | 65 | 5243 | active |
| katanaml/sparrow Sparrow is an open-source framework for structured data extraction from documents (PDFs, images) using ML, LLMs, and Vision LLMs, with sche… | 92 | 5225 | active |
| Kode AI Agent Platform Kode CLI is a terminal-based AI coding agent that understands codebases, edits files, runs commands, and automates developer workflows. It … | 60 | 5225 | active |
| BallonsTranslator A desktop GUI application that uses deep learning to automatically translate comics and manga, combining text detection, OCR, inpainting, a… | 98 | 5206 | active |
| Marker-Inc-Korea/AutoRAG AutoRAG 2.0 is a self-evolving librarian agent that searches local document collections (PDFs, wikis, notes) and curates results into struc… | 94 | 5163 | active |
| Magic MagiCrew (dtyq/magic) is an open-source, enterprise-grade all-in-one AI productivity platform combining a generalist AI agent, workflow eng… | 60 | 5049 | active |
| ministackorg/ministack MiniStack is a free, MIT-licensed local AWS emulator that mimics 60+ AWS services (S3, SQS, SNS, DynamoDB, Lambda, EC2, RDS, ElastiCache, E… | 79 | 4996 | active |
| LangWatch LangWatch is an open-source LLMOps platform for LLM evaluation, AI agent testing, tracing, and production observability. It combines agent … | 92 | 4988 | active |
| FellouAI/eko Eko is a production-ready JavaScript framework for building reliable AI agents and agentic workflows from natural language, running in both… | 57 | 4961 | active |
| campfirein/byterover-cli ByteRover CLI (brv) is a TypeScript-based interactive REPL/TUI tool that gives AI coding agents a persistent, structured memory layer via a… | 10 | 4958 | active |
| babaohuang/GeminiProChat A minimal web chat interface for Google's Gemini Pro API, built with TypeScript and Astro. It can be self-hosted via Vercel, Railway, Zeabu… | 58 | 4894 | active |
| ACE-Step ACE-Step is an open-source foundation model for music generation that combines diffusion-based generation with a deep compression autoencod… | 46 | 4881 | active |
| ethanniser/NextFaster NextFaster is a highly performant e-commerce template built with Next.js 15, featuring partial prerendering, Server Actions, Drizzle ORM on… | 64 | 4875 | active |
| eastlondoner/vibe-tools vibe-tools is a TypeScript CLI that gives AI coding agents like Cursor an 'AI team' of subagents and advanced skills such as browser automa… | 35 | 4822 | active |
| SeldonIO/seldon-core Seldon Core 2 is an MLOps and LLMOps framework for deploying, managing, and scaling machine learning models and modular AI applications on … | 66 | 4782 | active |
| openai/chatgpt-retrieval-plugin A FastAPI-based retrieval backend that provides semantic search over personal or organizational documents using natural language queries, w… | 31 | 21249 | maintenance |
| Kimi-Audio Kimi-Audio is an open-source audio foundation model (7B parameters) that unifies audio understanding, generation, and speech conversation i… | 28 | 4740 | active |
| VectifyAI/OpenKB OpenKB is an open-source CLI tool that compiles raw documents (PDF, Word, Markdown, HTML, and more) into a structured, interlinked wiki-sty… | 71 | 4717 | active |
| mindspore-ai/mindspore MindSpore is an open-source deep learning framework for training and inference across mobile, edge, and cloud scenarios. It provides automa… | 32 | 4714 | active |
| x-cmd/x-cmd X-CMD is a POSIX shell (bash/zsh/ash/dash) toolkit offering 300+ shell/awk modules and a 600+ curated CLI package system, installable witho… | 93 | 4695 | active |
| microsoft/PyRIT PyRIT is Microsoft's open-source Python framework for identifying security and safety risks in generative AI systems. It provides automatio… | 88 | 4674 | active |
| gradio-app/fastrtc FastRTC is a Python library that turns any Python function into a real-time audio and video stream over WebRTC or WebSockets. It includes b… | 52 | 4631 | active |
| joanrod/star-vector StarVector is a foundation model that generates scalable vector graphics (SVG) code from images and text by treating vectorization as a cod… | 46 | 4629 | active |
| Tencent-Hunyuan/HunyuanVideo-1.5 HunyuanVideo-1.5 is Tencent's lightweight 8.3B-parameter video generation model supporting text-to-video and image-to-video synthesis. The … | 48 | 4564 | active |
| TencentARC/InstantMesh InstantMesh is a feed-forward framework for generating 3D meshes from a single image using sparse-view large reconstruction models (LRM/Ins… | 25 | 4545 | active |
| JoshuaC215/agent-service-toolkit A full toolkit for building and running an AI agent service using LangGraph, FastAPI, and Streamlit. It provides a template including a Lan… | 68 | 4531 | active |