Open-source llm-inference software
3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
3358 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| llSourcell/Doctor-Dignity Doctor Dignity is a fine-tuned Llama2 7B model that can pass the US Medical Licensing Exam, built with PyTorch, Transformers, TRL, and ONNX… | 29 | 3809 | experimental |
| 0hq/WebGPT WebGPT is a vanilla JavaScript and HTML implementation of GPT transformer inference running in the browser via WebGPU, in under ~1500 lines… | 30 | 3787 | experimental |
| axtonliu/axton-obsidian-visual-skills A pack of prompt-based skills for Claude Code that generates Obsidian Canvas, Excalidraw, and Mermaid diagrams from text descriptions. It s… | 62 | 3639 | experimental |
| vercel-labs/fx fx is a tiny, open-source coding agent harness and CLI written in Zig, designed for minimalism, fast cold starts, and embeddability in larg… | 81 | 3535 | experimental |
| b-nnett/grok-bot-0.18-reconstructed An unofficial, source-oriented reconstruction of the Grok Bot 0.18.0 macOS Electron app, with readable TypeScript implementations of its ru… | 10 | 3521 | experimental |
| xjdr-alt/entropix Entropix is a research project implementing entropy-based sampling and parallel chain-of-thought decoding for large language models, aiming… | 23 | 3430 | experimental |
| NVIDIA-NeMo/Switchyard Switchyard is a Rust proxy and library that routes LLM traffic across models and providers while translating between OpenAI Chat, Anthropic… | 77 | 3421 | experimental |
| danielgross/localpilot A local proxy server that emulates GitHub Copilot by running open-source code-completion LLMs (via llama.cpp/GGML) on a Mac, so VS Code's C… | 28 | 3337 | experimental |
| oboard/claude-code-rev A restored, runnable source tree of Anthropic's Claude Code CLI, reconstructed from source maps with shims filling unrecoverable modules. I… | 55 | 3300 | experimental |
| microsoft/amplifier Amplifier is a modular, extensible AI-powered development assistant delivered as a command-line tool from Microsoft. It provides chat and a… | 63 | 3126 | experimental |
| evilsocket/cake Cake is a multimodal AI inference server written in Rust that runs text, image, and voice models on a single device or shards them across a… | 56 | 3126 | experimental |
| daveshap/OpenAI_Agent_Swarm A framework for building a Hierarchical Autonomous Agent Swarm (HAAS) on top of OpenAI's agent APIs, where a core set of governed agents ca… | 10 | 3097 | experimental |
| OpenNSWM-Lab/FAROS FAROS is a blueprint-driven AutoResearch runtime that orchestrates end-to-end AI research workflows, from idea generation through experimen… | 56 | 3037 | experimental |
| Yeachan-Heo/gajae-code Gajae-Code (gjc) is an external coding-agent harness that runs AI coding agents in any repository using the coding subscription plans you a… | 77 | 3018 | experimental |
| RootbeerComputer/backend-GPT An experimental project that replaces a traditional backend and database with an LLM. Users describe their app's purpose, provide an initia… | 32 | 2931 | experimental |
| wquguru/nof0 NOF0 is an open-source AI trading arena that lets multiple LLM/agent-driven strategies compete in live cryptocurrency markets, each startin… | 39 | 2749 | experimental |
| gvzdv/claudish-to-english A Claude Code plugin that rewrites assistant messages into plain English using a local LLM via ollama, or alternatively the codex CLI, Anth… | 77 | 2728 | experimental |
| ishan0102/vimGPT vimGPT is a Python application that lets GPT-4V browse and interact with the web visually using Playwright and the Vimium keyboard-navigati… | 28 | 2643 | experimental |
| mshumer/gpt-author A Jupyter notebook-based tool that chains GPT-4/Claude 3 and Stable Diffusion API calls to generate complete fantasy novels from a user pro… | 30 | 2530 | experimental |
| semanser/codel Codel is a self-hosted, fully autonomous AI agent that executes complex tasks using a sandboxed Docker environment with terminal, browser, … | 16 | 2476 | experimental |
| IlyaRice/RAG-Challenge-2 A Python implementation of the winning RAG system from the Enterprise RAG Challenge 2 competition, answering questions about company annual… | 26 | 2458 | experimental |
| jennyzzt/dgm Darwin Gödel Machine (DGM) is a research implementation of a self-improving AI agent that iteratively rewrites its own codebase and validat… | 32 | 2424 | experimental |
| keijiro/AIShader A Unity editor plugin that generates shaders from prompts using the ChatGPT API. It is a proof-of-concept tool demonstrating LLM-driven sha… | 31 | 2381 | experimental |
| Autodoc Autodoc is an experimental CLI toolkit that auto-generates documentation for git repositories using large language models like GPT-4. It in… | 31 | 2373 | experimental |
| dvmazur/mixtral-offloading A Python library enabling efficient inference of Mixtral-8x7B mixture-of-experts language models on limited hardware like Google Colab or c… | 27 | 2334 | experimental |
| Sumandora/remove-refusals-with-transformers A proof-of-concept Python script that removes refusal behavior from LLMs by ablating the refusal direction using pure Hugging Face Transfor… | 46 | 2230 | experimental |
| emcf/engshell Engshell is an English-language shell powered by LLMs that translates natural-language commands into executable Python code on any OS. It i… | 57 | 2145 | experimental |
| etched-ai/open-oasis Inference code and model weights for Oasis 500M, an interactive world model from Decart and Etched that generates gameplay video autoregres… | 22 | 2145 | experimental |
| YiVal/YiVal YiVal is an open-source Python framework that automatically tunes prompts, RAG configurations, and model parameters for generative AI appli… | 20 | 2137 | experimental |
| renatoasse/opensquad opensquad is a multi-agent orchestration framework that lets users describe tasks in natural language and automatically assembles teams of … | 46 | 2120 | experimental |
| Lightricks/LTX-Desktop LTX Desktop is an open-source desktop application for generating videos with Lightricks' LTX generative video models, either locally on sup… | 83 | 2112 | experimental |
| fikrikarim/parlor Parlor is a fully on-device, real-time multimodal voice assistant similar to GPT-Live, combining speech recognition, a Gemma vision-languag… | 73 | 2076 | experimental |
| microsoft/aici AICI (Artificial Intelligence Controller Interface) is a Microsoft Research prototype that lets developers write Wasm-based controllers to … | 19 | 2076 | experimental |
| fiatrete/OpenDAN-Personal-AI-OS OpenDAN is an open-source Personal AI Operating System that consolidates various AI modules into one platform for personal use. It lets use… | 50 | 2061 | experimental |
| Ante Ante is a self-contained terminal-based coding agent harness written in Rust, distributed as a single ~15MB binary with zero runtime depend… | 80 | 2011 | experimental |
| browser-use/macOS-use A Python library that lets LLM-powered AI agents control and interact with any macOS application via the accessibility layer, from the make… | 10 | 2002 | experimental |
| Mini-Omni Mini-Omni is an open-source multimodal large language model that performs real-time end-to-end speech-to-speech conversation with streaming… | 23 | 1935 | experimental |
| losfair/blueboat Blueboat is an all-in-one, multi-tenant serverless JavaScript runtime written in Rust, designed for frictionless horizontal scalability. It… | 23 | 1923 | experimental |
| Elysia Elysia is an agentic platform and Python package from Weaviate where a decision agent dynamically chooses tools arranged in a decision tree… | 51 | 1919 | experimental |
| StartupHakk/OpenMonoAgent.ai OpenMonoAgent is a terminal-native coding agent that runs entirely on local hardware, pairing a .NET 10 CLI with a bundled llama.cpp infere… | 57 | 1866 | experimental |
| jina-ai/dev-gpt Dev-GPT is a Python CLI tool that simulates a virtual development team (Product Manager, Developer, DevOps) powered by OpenAI models. Users… | 31 | 1866 | experimental |
| Standard-Intelligence/hertz-dev Hertz-dev is an open-source 8.5B parameter autoregressive base model for full-duplex conversational audio, released by Standard Intelligenc… | 22 | 1799 | experimental |
| zhnt/loushang Loushang is an AI-native agent harness for coding workflows, delivered as a Python CLI and terminal workbench (`loushang code`) with multi-… | 59 | 1758 | experimental |
| antirez/voxtral.c A pure C, zero-dependency inference implementation of Mistral's Voxtral Realtime 4B speech-to-text model, with a streaming C API and CLI fo… | 42 | 1750 | experimental |
| lidangzzz/goal-driven Goal-Driven is a prompt-based multi-agent methodology that wraps coding agents like Claude Code or Codex in a master-agent loop, keeping su… | 44 | 1744 | experimental |
| cfortuner/promptable Promptable is a TypeScript/JavaScript library for building LLM-powered applications, providing prompt templating, model providers (OpenAI, … | 10 | 1744 | experimental |
| Cormanz/smartgpt SmartGPT is an experimental Rust program that lets LLMs like GPT-3.5 and GPT-4 autonomously complete complex tasks by breaking them into su… | 31 | 1737 | experimental |
| vercel-labs/openreview OpenReview is an open-source, self-hosted AI code review bot that deploys to Vercel and connects to GitHub via a GitHub App. Mentioning @op… | 43 | 1696 | experimental |
| skyllwt/AutoSci AutoSci is a wiki-centric AI research platform that automates the full scientific research lifecycle — reading literature, thinking, runnin… | 58 | 1689 | experimental |
| eylonmiz/react-agent ReactAgent is an open-source autonomous LLM agent that uses GPT-4 to generate and compose React components from user stories. It is built w… | 21 | 1656 | experimental |
| deepseek-ai/DeepSeek-V3.2-Exp DeepSeek-V3.2-Exp is an experimental open-weight large language model release that introduces DeepSeek Sparse Attention for more efficient … | 37 | 1654 | experimental |
| SETA OWL is a Python‑based framework that enables multiple AI agents to collaborate on real‑world task automation. It builds on the CAMEL‑AI eco… | 60 | 179 | active |
| collabora/WhisperFusion WhisperFusion is a real-time voice chat application that combines WhisperLive speech-to-text, a Mistral/Phi LLM, and WhisperSpeech text-to-… | 27 | 1648 | experimental |
| cyberpapiii/chipotlai-max Chipotlai Max is a meme fork of the OpenCode AI coding agent that hardcodes Chipotle's reverse-engineered 'Pepper' support chatbot as the d… | 50 | 1572 | experimental |
| ghuntley/loom Loom is an AI-powered coding agent written in Rust that provides a REPL interface for interacting with LLM-driven agents capable of file sy… | 61 | 1567 | experimental |
| protectai/rebuff Rebuff is a self-hardening prompt injection detector that protects LLM-powered applications through multi-layered defenses: heuristics, a d… | 10 | 1521 | experimental |
| YGYOOO/WorldX WorldX is an open-source web application that generates an AI-driven virtual world from a single sentence, creating maps, pixel-art charact… | 58 | 1512 | experimental |
| daveshap/ACE_Framework ACE (Autonomous Cognitive Entities) is an open-source framework for building fully autonomous AI agents that run entirely on local hardware… | 10 | 1501 | experimental |
| murtaza-nasir/maestro MAESTRO is a self-hosted AI-powered research platform that plans and executes complex research tasks using AI agents. It generates detailed… | 54 | 1492 | experimental |
| Learning to (Learn at Test Time) Language Model Official PyTorch implementation of Test-Time Training (TTT) layers, a sequence modeling layer with linear complexity whose hidden state is … | 24 | 1405 | experimental |
| jdagdelen/hyperDB HyperDB is a lightweight, local vector database library for Python with a C++/MKL-accelerated backend, designed to store and query document… | 21 | 1401 | experimental |
| Parallax Parallax is a distributed model serving framework that lets you build your own AI cluster across heterogeneous, geographically dispersed de… | 61 | 1397 | experimental |
| ProjectUnifree/unifree A Python CLI tool that helps migrate Unity game projects to other engines, primarily Godot and Unreal Engine. It scans Unity projects for m… | 28 | 1397 | experimental |
| socketteer/loom Loom is an experimental tree-based writing interface for collaborating with GPT-3, letting users explore branching continuations of a text … | 32 | 1390 | experimental |
| AutoArk/TinyEngram TinyEngram is an open research project exploring DeepSeek-AI's Engram architecture and memory injection as an alternative to LoRA for param… | 50 | 1380 | experimental |
| gorilla-llm/gorilla-cli Gorilla CLI is a command-line tool that turns plain-English objectives into candidate shell commands using LLMs like Gorilla LLM, GPT-4, an… | 20 | 1374 | experimental |
| OpenBB-finance/experimental-openbb-platform-agent A Python library and R&D playground from OpenBB that builds LLM-powered financial analyst agents on top of the OpenBB Platform. Agents use … | 18 | 1348 | experimental |
| linyiLYi/voice-assistant A simple single-script Python demo of a local voice assistant that uses Whisper (via Apple MLX) for speech recognition and a local Yi large… | 27 | 1323 | experimental |
| Claw-AI-Lab/Claw-AI-Lab Claw AI Lab is a self-hosted multi-agent platform that simulates a full AI research lab from a single prompt, orchestrating specialized age… | 51 | 1315 | experimental |
| DLYuanGod/TinyGPT-V TinyGPT-V is an efficient multimodal large language model built on small backbones (Phi-2 2.7B), combining vision and language capabilities… | 53 | 1314 | experimental |
| anc95/writely Writely is an open-source browser extension for Chrome, Firefox, and Edge that brings GPT-powered AI writing assistance to any editable web… | 33 | 1296 | experimental |
| microsoft/prompty Prompty is a markdown-based file format (.prompty) for authoring LLM prompts with YAML frontmatter for model config, inputs, tools, and tem… | 91 | 1285 | experimental |
| Runfusion/Fusion Fusion is an open-source multi-agent orchestrator that acts as a 'software factory': AI agents plan, build, review, and ship code from plai… | 80 | 1280 | experimental |
| nethical6/conversation-steganography A Go CLI tool that hides encrypted secret messages inside natural-looking chat text generated by a local LLM (GPT-2), enabling covert commu… | 52 | 1250 | experimental |
| rapidpages/rapidpages Rapidpages is a prompt-first IDE that generates React and Tailwind CSS components from natural language descriptions using AI. It can be se… | 28 | 1232 | experimental |
| NoDataFound/hackGPT hackGPT is a Python/Jupyter-based chatbot application that leverages OpenAI and ChatGPT APIs for security-oriented tasks like malware analy… | 72 | 1212 | experimental |
| soongenwong/claudecode An open-source, Rust-based terminal AI coding agent inspired by Anthropic's Claude Code, offering an interactive agent shell, one-shot prom… | 51 | 1207 | experimental |
| Ido-Levi/Hephaestus Hephaestus is a Python-based semi-structured agentic framework where AI agents dynamically create tasks and build workflows as they discove… | 33 | 1185 | experimental |
| CSCB/vibe-mouse An open-source Python desktop application that redefines mouse interaction by letting users bind customizable 'Skills' to buttons, with mul… | 52 | 1179 | experimental |
| farzaa/gemini-bball A demo project from a viral tweet that uses Google's Gemini API to analyze basketball video frames, with an OpenCV-based visualizer. The co… | 29 | 1163 | experimental |
| elfvingralf/macOSpilot-ai-assistant macOSpilot is a macOS desktop AI assistant built with Electron that answers spoken or typed questions about whatever application is current… | 27 | 1158 | experimental |
| berylliumsec/nebula Nebula is an AI-powered penetration testing desktop application that combines a terminal, browser, notes, findings, and reporting into one … | 96 | 1128 | experimental |
| Axflow Axflow is a modular TypeScript framework for building natural language powered AI applications, composed of packages like @axflow/models (S… | 20 | 1122 | experimental |
| microsoft/agentrc AgentRC is a Microsoft CLI tool (also a VS Code extension) that measures how 'AI-ready' a repository is and generates context files like Co… | 70 | 1107 | experimental |
| oxylabs/ai-scraper-py AI-Scraper is a Python library and scrape agent from Oxylabs AI Studio that extracts data from webpages using natural language prompts inst… | 10 | 1090 | experimental |
| agents-io/PokeClaw PokeClaw is an open-source Android app that runs an on-device LLM (Gemma 4 via LiteRT) to control and automate your phone, with no cloud or… | 65 | 1083 | experimental |
| thinking-machines-lab/batch_invariant_ops A Python library that replaces standard PyTorch CUDA kernels with batch-invariant versions, ensuring identical results regardless of batch … | 37 | 1080 | experimental |
| srush/llama2.rs A pure Rust implementation of Llama2 inference optimized for fast CPU execution, featuring 4-bit GPT-Q quantization, SIMD support, and memo… | 29 | 1064 | experimental |
| yoheinakajima/ditto Ditto is a simple self-building coding agent that generates multi-file Flask applications from natural language descriptions via a web inte… | 23 | 1064 | experimental |
| Omniplex-ai/omniplex Omniplex is an open-source, self-hosted alternative to Perplexity, an AI-powered answer engine that combines web search with LLMs to genera… | 25 | 1060 | experimental |
| ddupont808/GPT-4V-Act GPT-4V-Act is a multimodal AI agent that combines GPT-4V(ision) with a Chromium browser, using Set-of-Mark prompting and a DOM auto-labeler… | 28 | 1059 | experimental |
| huggingface/optimum-nvidia Optimum-NVIDIA is a Python library that bridges Hugging Face Transformers with NVIDIA TensorRT-LLM for highly optimized LLM inference on NV… | 62 | 1039 | experimental |
| trotsky1997/MathBlackBox MathBlackBox (MCTSr) is a Python research implementation of Monte Carlo Tree Self-Refine, an algorithm that combines MCTS with LLM self-ref… | 24 | 1032 | experimental |
| orca-wm/Orca Orca is a general world foundation model from BAAI centered on Next-State-Prediction, learning a unified world latent space from visual and… | 54 | 1021 | experimental |
| susiai/susi_shell A suite of Python-based command line tools for interacting with AI services directly from the terminal, including chat, text completion, tr… | 37 | 1000 | experimental |
| meta-llama/llama Meta's official repository containing inference code and download scripts for Llama 2 large language models (7B to 70B parameters). The rep… | 31 | 59632 | abandoned |
| continuedev/continue Continue is an open-source AI coding agent distributed as a CLI, VS Code extension, and JetBrains plugin, offering agent mode, chat, edit, … | 86 | 36172 | abandoned |
| Pythagora-io/gpt-pilot GPT Pilot was an AI coding agent that aimed to act as an 'AI developer', generating full applications step-by-step from a specification usi… | 65 | 33662 | abandoned |
| voideditor/void Void is an open-source AI code editor forked from VS Code, offering chat, agent mode, inline edits, and autocomplete with any LLM provider … | 10 | 28781 | abandoned |
| openai/gpt-2 Official OpenAI repository containing code and pretrained model weights for GPT-2, from the paper 'Language Models are Unsupervised Multita… | 10 | 25009 | abandoned |