Open-source llm-inference software
3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
3358 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| 0xShug0/audio.cpp audio.cpp is a pure C++ inference engine for audio models built on ggml, supporting TTS, STT, VAD, voice conversion, music generation, and … | 82 | 3646 | active |
| mit-han-lab/llm-awq AWQ is a Python library implementing Activation-aware Weight Quantization for compressing large language models to low-bit (INT3/4) weights… | 41 | 3638 | active |
| NExT-GPT/NExT-GPT NExT-GPT is an end-to-end any-to-any multimodal large language model that accepts and generates arbitrary combinations of text, image, vide… | 34 | 3638 | active |
| simonlin1212/TradingAgents-astock A Python multi-agent investment research framework for China's A-share market, deeply forked from TauricResearch/TradingAgents. It runs 7 s… | 80 | 3635 | active |
| AIDotNet/OpenDeepWiki OpenDeepWiki is an AI-driven platform that turns Git repositories, ZIP archives, and local directories into searchable, structured document… | 85 | 3615 | active |
| nv-tlabs/kimodo Kimodo is NVIDIA's official implementation of a kinematic motion diffusion model trained on 700 hours of motion capture data to generate hi… | 58 | 3615 | active |
| Qwen3-ASR Qwen3-ASR is a family of open-source speech recognition models from Alibaba's Qwen team, supporting ASR and language identification across … | 52 | 3605 | active |
| HolmesGPT/holmesgpt HolmesGPT is an open-source AI agent (CNCF Sandbox project) that investigates production incidents and finds root causes by querying live o… | 88 | 3604 | active |
| deta/surf Deta Surf is an open-source, local-first AI notebook for macOS, Windows, and Linux that lets users organize files, PDFs, websites, and YouT… | 72 | 3600 | active |
| xororz/local-dream A free, open-source Android app for running Stable Diffusion locally with Snapdragon NPU acceleration, also supporting CPU/GPU inference. I… | 82 | 3598 | active |
| HanaokaYuzu/Gemini-API A reverse-engineered asynchronous Python client library for Google's Gemini web app (formerly Bard), published as gemini-webapi on PyPI. It… | 90 | 3597 | active |
| RUC-NLPIR/FlashRAG FlashRAG is a modular Python toolkit for reproducing and developing Retrieval-Augmented Generation (RAG) research, bundling 36 pre-processe… | 88 | 3590 | active |
| whoiskatrin/chart-gpt Chart-GPT is a web application that generates charts from natural language text input using AI. Users describe what they want to visualize … | 39 | 3581 | active |
| ChatGPT Java SDK An unofficial Java SDK that wraps OpenAI's ChatGPT API for GPT-3.5, GPT-4, GPT-4o, and GPT-5 models, published on Maven Central with a flue… | 44 | 3579 | active |
| liu-ziting/what-to-eat An AI-powered recipe generation web platform built with Vue 3 and TypeScript that creates recipes across China's eight major cuisines plus … | 42 | 3567 | active |
| CosmosShadow/gptpdf A small Python library that parses PDF files into Markdown using a vision-capable LLM such as GPT-4o. It uses PyMuPDF to detect non-text ar… | 28 | 3559 | active |
| SkyworkAI/DeepResearchAgent DeepResearchAgent is a Python framework for hierarchical multi-agent systems that combines a top-level planning agent with specialized lowe… | 63 | 3552 | active |
| liujuntao123/smart-excalidraw-next A web-based AI-powered diagramming application built on Excalidraw that generates professional charts and diagrams from natural language de… | 62 | 3546 | active |
| microsoft/hummingbird Hummingbird is a Microsoft library that compiles trained traditional ML models (decision trees, random forests, XGBoost, LightGBM) into ten… | 65 | 3545 | active |
| rashadphz/farfalle Farfalle is an open-source, self-hosted AI-powered search engine that works like Perplexity, answering questions with cited web search resu… | 25 | 3543 | active |
| fnnx-ai/scikit-llm Scikit-LLM is a Python library that integrates large language models like GPT into scikit-learn workflows as estimators. It enables tasks s… | 72 | 3533 | active |
| google-deepmind/alphafold Open-source implementation of the AlphaFold 2 inference pipeline for predicting protein structures from amino acid sequences, including Alp… | 54 | 14887 | maintenance |
| Mesh-LLM/mesh-llm Mesh LLM is a Rust-based distributed LLM inference runtime that pools GPUs and memory across machines into a single OpenAI-compatible API. … | 79 | 3522 | active |
| EverMind-AI/MSA MSA (Memory Sparse Attention) is a Python framework for end-to-end trainable sparse latent-memory attention that scales LLM context to 100M… | 50 | 3520 | active |
| google-ai-edge/LiteRT LiteRT is Google's successor to TensorFlow Lite, an on-device runtime for high-performance ML and GenAI inference on edge platforms. It pro… | 86 | 3510 | active |
| PKU-YuanGroup/Video-LLaVA Video-LLaVA is a large vision-language model that aligns image and video representations into a unified visual space before projection into… | 28 | 3504 | active |
| huggingface/optimum Optimum is a Hugging Face library that extends Transformers, Diffusers, timm, and Sentence Transformers with hardware-specific optimization… | 91 | 3503 | active |
| openai/openai-go The official Go client library for the OpenAI REST API, providing typed access to models, the Responses API, conversations, and other OpenA… | 87 | 3501 | active |
| JusticeRage/Gepetto Gepetto is an IDA Pro plugin that queries large language models (OpenAI, Anthropic, Gemini, Ollama, and others) to explain decompiled funct… | 73 | 3487 | active |
| FireRedTeam/FireRed-OpenStoryline FireRed-OpenStoryline is an AI video editing agent that converts natural language instructions into directed video edits via LLM planning a… | 55 | 3483 | active |
| fauxpilot/fauxpilot FauxPilot is a self-hosted, open-source alternative to the GitHub Copilot code completion server. It serves SalesForce CodeGen models via N… | 32 | 14676 | maintenance |
| imraywang/wewrite WeWrite is a Python-based AI agent skill that automates the full WeChat Official Account content pipeline: topic selection from trending ne… | 77 | 3473 | active |
| deepseek-ai/DeepSeek-OCR-2 DeepSeek-OCR 2 is an open-source vision-language model and inference toolkit implementing 'Visual Causal Flow' for optical character recogn… | 42 | 3471 | active |
| Linly Digital Human System Linly-Talker is a Python-based digital avatar conversational system that combines LLMs (Linly, Qwen, Gemini), speech recognition (Whisper, … | 44 | 3468 | active |
| MiniMax AI Platform Official repository for MiniMax-Text-01 and MiniMax-VL-01, open-weight large language and vision-language models built on a linear attentio… | 31 | 3466 | active |
| n3d1117/chatgpt-telegram-bot A self-hosted Telegram bot written in Python that integrates with OpenAI's official ChatGPT, DALL·E, and Whisper APIs to answer questions, … | 30 | 3464 | active |
| lynote-ai/humanize-text An open-source Python toolkit for making AI-generated drafts read more naturally, offering four documented humanization approaches plus a p… | 67 | 3453 | active |
| entropy-research/Devon Devon is an open-source AI pair programmer that acts as an autonomous software engineering agent, installable via pipx (backend) and npm (U… | 32 | 3450 | active |
| Mirix-AI/MIRIX MIRIX is a self-hosted multi-agent personal assistant that captures on-screen activity and conversation to build structured long-term memor… | 70 | 3445 | active |
| i-am-bee/beeai-framework BeeAI Framework is an open-source, Linux Foundation-hosted framework for building production-grade multi-agent AI systems in Python and Typ… | 86 | 3436 | active |
| alibaba/ROLL ROLL is an open-source reinforcement learning library from Alibaba for training large language models at scale, supporting algorithms like … | 82 | 3429 | active |
| RKNN-Toolkit2 RKNN-Toolkit2 is Rockchip's SDK for converting trained neural network models into RKNN format and deploying them on Rockchip NPU chips like… | 33 | 3419 | active |
| Clowder AI Clowder AI is a TypeScript framework for building teams of AI agents that collaborate across multiple models like Claude, GPT, and Gemini i… | 81 | 3418 | active |
| chatgpt-java An unofficial Java SDK for the OpenAI API covering all official endpoints including chat completions (GPT-3.5/GPT-4), DALL-E image generati… | 22 | 3406 | active |
| Intelligent-Internet/ii-agent II-Agent is an open-source (Apache-2.0) AI agent framework for building and deploying intelligent assistants that execute complex multi-ste… | 62 | 3391 | active |
| pashpashpash/vault-ai OP Vault is a self-hosted application combining a Go backend with a React frontend that lets users upload documents (PDF, txt, epub) into a… | 40 | 3389 | active |
| edison7009/EchoBird EchoBird is a cross-platform desktop application (Tauri + Rust) that provides one-click installation, launching, and model configuration fo… | 79 | 3387 | active |
| LangGraph.js LangGraph.js is a TypeScript/JavaScript framework for building stateful, long-running LLM agents as graphs of nodes and edges, mixing deter… | 90 | 3366 | active |
| ANative-Lab/EvoAgentX EvoAgentX is an open-source Python framework for building, evaluating, and automatically evolving LLM-based agents and agentic workflows. I… | 78 | 3363 | active |
| ColeMurray/background-agents An open-source, self-hostable background coding agent system (Open-Inspect) inspired by Ramp's Inspect, where AI agents work on tasks in sa… | 61 | 3363 | active |
| OpenGVLab/Ask-Anything VideoChat/Ask-Anything is a family of multimodal chat models and demos that combine video understanding with large language models, letting… | 69 | 3354 | active |
| Lagrange-Labs/deep-prove DeepProve is a Rust framework for generating zero-knowledge proofs of neural network inference, including end-to-end proving of full LLM fo… | 57 | 3352 | active |
| Sophomoresty/gemini-web2api A single-file Python server that exposes Google Gemini's web interface as an OpenAI-compatible API, supporting chat completions, streaming,… | 54 | 3346 | active |
| nicedreamzapp/claude-code-local An MLX-native server for Apple Silicon that exposes an Anthropic-API-compatible endpoint so Claude Code and similar agents can run entirely… | 75 | 3344 | active |
| fuxicodex/Fuxi FuXi is a terminal-based AI coding agent built in Go that reads code, edits files, runs commands, and drives tools through a rich TUI using… | 78 | 3341 | active |
| QiuYannnn/Local-File-Organizer A privacy-focused desktop application that uses on-device AI models (Llama3.2 3B and Llava v1.6 via Nexa SDK) to scan, understand, and reor… | 14 | 3340 | active |
| ResearAI/DeepScientist DeepScientist is a local-first autonomous research studio that runs the full scientific research loop on your machine, from baselines and e… | 68 | 3335 | active |
| Bytez-com/docs Bytez is a serverless model inference platform offering a unified API with a single key to run 175k+ open and closed-source AI models (LLMs… | 53 | 3329 | active |
| AtomicBot-ai/atomic-agent Atomic Agent is a local-first AI agent that runs entirely on your machine, using quantized models via llama.cpp to browse the web, edit fil… | 78 | 3326 | active |
| LLM-as-a-Verifier A Python framework that uses LLM token logprobs to produce fine-grained verification scores for agent trajectories without additional train… | 59 | 3322 | active |
| johnbean393/Sidekick Sidekick is a native macOS app for chatting with local LLMs powered by a built-in llama.cpp backend, with RAG over local files, folders, an… | 75 | 3320 | active |
| Mercury Agent Mercury Agent is a soul-driven AI assistant that runs 24/7 from the CLI, Telegram, Discord, Slack, Signal, or web, with 40+ permission-hard… | 81 | 3315 | active |
| wendy7756/AI-Video-Transcriber An open-source AI tool that transcribes, summarizes, and archives videos and podcasts from 30+ platforms (YouTube, TikTok, Bilibili, etc.) … | 61 | 3314 | active |
| cita-777/metapi Metapi is a self-hosted meta-aggregation gateway that unifies multiple AI relay/proxy sites (New API, One API, OneHub, DoneHub, Veloera, et… | 70 | 3308 | active |
| ikawrakow/ik_llama.cpp A fork of llama.cpp offering state-of-the-art quantization types and improved CPU/CUDA inference performance for large language models. It … | 60 | 3304 | active |
| itsOwen/CyberScraper-2077 CyberScraper 2077 is an AI-powered web scraping application with a Streamlit GUI that uses OpenAI, Gemini, or local Ollama models to intell… | 68 | 3291 | active |
| jehna/humanify A Rust CLI tool that unminifies and deobfuscates JavaScript by using LLMs (OpenAI, Anthropic, Gemini, OpenRouter, or local Ollama) to sugge… | 83 | 3288 | active |
| PrismML-Eng/Bonsai-demo A demo repository for running PrismML's Bonsai family of 1-bit and ternary-quantized language models locally via llama.cpp and MLX. It prov… | 59 | 3286 | active |
| KouriChat/KouriChat KouriChat is a Python-based emotional companionship program powered by LLMs that lets users chat with realistic AI personas. It integrates … | 85 | 3281 | active |
| microsoft/lida LIDA is a Python library from Microsoft that uses large language models to automatically generate, execute, edit, explain, evaluate, and re… | 31 | 3279 | active |
| gptscript-ai/gptscript GPTScript is a Go-based framework that lets LLMs operate and interact with systems such as local executables, OpenAPI endpoints, SDKs, and … | 10 | 3279 | active |
| pezzolabs/pezzo Pezzo is an open-source, cloud-native LLMOps platform for managing the lifecycle of AI-powered features, including prompt design, versionin… | 62 | 3275 | active |
| zai-org/ChatGLM3 ChatGLM3 is a series of open bilingual (Chinese-English) conversational large language models from Zhipu AI and Tsinghua University, includ… | 28 | 13646 | maintenance |
| aandrew-me/tgpt tgpt is a cross-platform command-line tool written in Go that lets users interact with AI chatbots (GPT, Llama, Mixtral, etc.) directly fro… | 96 | 3271 | active |
| op7418/NanoBanana-PPT-Skills An AI-powered tool that generates high-quality presentation slides as images using Google's Nano Banana Pro (Gemini image model), with opti… | 41 | 3266 | active |
| gcui-art/suno-api An unofficial open-source API wrapper for Suno.ai's AI music generation service, exposing endpoints for generating songs, lyrics, and audio… | 53 | 3260 | active |
| SawyerHood/draw-a-ui A Next.js web app that lets users draw UI wireframes on a tldraw canvas and generates HTML with Tailwind CSS from the sketch using the GPT-… | 39 | 13583 | maintenance |
| eracle/OpenOutreach OpenOutreach is a self-hosted, open-source AI agent for B2B lead generation. You describe your product and target market, and it discovers … | 77 | 3254 | active |
| jjyaoao/HelloAgents HelloAgents is a Python multi-agent framework built on OpenAI-compatible APIs, offering tool response protocols, context engineering, sessi… | 84 | 3250 | active |
| cursor-byok cursor-byok is an open-source local implementation of Cursor's backend that runs a gateway service on your machine, connecting the Cursor e… | 77 | 3246 | active |
| Harbor Harbor is a CLI tool that spins up a complete, pre-wired local LLM stack with a single command, using Docker Compose to orchestrate hundred… | 84 | 3241 | active |
| MisoLabsAI/MisoTTS Miso TTS 8B is an open-source text-to-speech model based on an RVQ Transformer architecture with a Llama 3.2-style 8B backbone, designed fo… | 49 | 3240 | active |
| Jittor/jittor Jittor is a high-performance deep learning framework from Tsinghua University based on just-in-time (JIT) compilation and meta-operators, w… | 68 | 3237 | active |
| zai-org/GLM-4-Voice GLM-4-Voice is an end-to-end bilingual (Chinese/English) speech dialogue model from Zhipu AI, built on GLM-4-9B with a speech tokenizer and… | 22 | 3237 | active |
| Off Grid AI Off Grid AI (OGAM) is a cross-platform mobile and desktop application that runs AI entirely on-device: GGUF LLM chat with vision, Whisper s… | 78 | 3234 | active |
| guaguaguaxia/weekly_report A web application that generates complete weekly work reports from brief descriptions of work content, powered by the OpenAI GPT-3.5 API. I… | 74 | 3231 | active |
| google-gemini/computer-use-preview A Python reference implementation of Gemini's computer-use agent that lets the model control a browser (via Playwright or Browserbase) to c… | 59 | 3224 | active |
| alexrudall/ruby-openai A Ruby gem providing a client for the OpenAI API, supporting chat completions, the Responses API, streaming, vision, audio (Whisper), image… | 66 | 3223 | active |
| Alpha-Dojo/DojoAgents DojoAgents is a full-market AI copilot framework for personal investment, built around a Loop-Driven Cognitive Portfolio Agent that reasons… | 57 | 3221 | active |
| qdrant/fastembed FastEmbed is a lightweight Python library for generating text embeddings using ONNX Runtime instead of PyTorch, requiring no GPU and minima… | 90 | 3216 | active |
| Josh-XT/AGiXT AGiXT is an open-source AI agent automation platform that orchestrates instruction management and complex task execution across multiple AI… | 82 | 3216 | active |
| crawlab-team/artipub ArtiPub is a self-hosted article publishing platform that automatically distributes content to multiple media channels such as CSDN and Seg… | 48 | 3205 | active |
| ExplosiveCoderflome/AI-Novel-Writing-Assistant An AI-native open-source workspace for producing complete long-form novels, orchestrating planning, worldbuilding, character assets, chapte… | 84 | 3202 | active |
| SuperCmdLabs/SuperCmd SuperCmd is an open-source macOS launcher combining Raycast-compatible extensions, hold-to-speak dictation, text-to-speech, AI chat and age… | 69 | 3202 | active |
| xiamuceer-j/MuMuAINovel MuMuAINovel is a self-hosted AI-powered novel writing assistant built with FastAPI and React. It helps authors generate outlines, character… | 85 | 3188 | active |
| matthartman/ghost-pepper A free, open-source macOS menu bar app providing fully on-device speech-to-text dictation and meeting transcription using local Whisper, Pa… | 72 | 3186 | active |
| MiniMax AI Platform MiniMax-M1 is an open-weight, large-scale hybrid-attention reasoning language model released by MiniMax under Apache-2.0. The repository pr… | 29 | 3185 | active |
| CatchTheTornado/text-extract-api A self-hosted FastAPI-based API that converts PDFs, Office documents, and images into Markdown or structured JSON using OCR engines (EasyOC… | 42 | 3183 | active |
| PenglongHuang/chinese-novelist-skill A Claude Code skill that drives AI to write complete Chinese novels end-to-end. It uses layered interactive Q&A, cross-session preference m… | 58 | 3179 | active |
| tekaratzas/RustGPT A transformer-based large language model implemented entirely in pure Rust with no external ML frameworks, using only ndarray for matrix op… | 35 | 3175 | active |