Open-source llm-inference software
3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
3358 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| Roo Code Roo Code is a VS Code extension that provides a team of AI coding agents inside the editor, with modes for coding, architecture planning, d… | 10 | 24296 | abandoned |
| openai/swarm Swarm is an experimental, educational Python framework from OpenAI for lightweight multi-agent orchestration, built on the Chat Completions… | 57 | 22004 | abandoned |
| QwenLM/Qwen The official repository for Qwen (Tongyi Qianwen), Alibaba Cloud's family of pretrained and chat large language models, with weights on Hug… | 57 | 21858 | abandoned |
| opencode-ai/opencode OpenCode is a terminal-based AI coding agent written in Go, providing a TUI for interacting with multiple AI providers (OpenAI, Anthropic, … | 10 | 13767 | abandoned |
| fuergaosi233/wechat-chatgpt A TypeScript bot that connects OpenAI's ChatGPT API to WeChat using the wechaty library, supporting text conversations, DALL·E image genera… | 23 | 13229 | abandoned |
| idootop/mi-gpt MiGPT is a Node.js/Docker application that connects Xiaomi's XiaoAI smart speakers to ChatGPT and Doubao, turning them into custom AI voice… | 10 | 12499 | abandoned |
| mistralai/mistral-inference The official minimal Python library from Mistral AI for running inference with Mistral open-weight models such as Mistral 7B, Mixtral, Code… | 10 | 10820 | abandoned |
| GitHub Copilot Chat The GitHub Copilot Chat extension for Visual Studio Code, providing AI chat, inline suggestions, and autonomous agent capabilities inside t… | 10 | 9960 | abandoned |
| aliasrobotics/cai Cybersecurity AI (CAI) is an open-source Python framework of specialized AI agents for offensive security tasks such as penetration testing… | 10 | 9845 | abandoned |
| intel/ipex-llm IPEX-LLM is a PyTorch LLM acceleration library for Intel hardware (iGPU, NPU, Arc/Flex/Max GPUs, and CPU), offering low-bit quantization (F… | 10 | 8845 | abandoned |
| OptiMate OptiMate is a collection of Python libraries from Nebuly AI for optimizing AI model performance, including Speedster for inference accelera… | 23 | 8328 | abandoned |
| EleutherAI/gpt-neo GPT-Neo is EleutherAI's implementation of model- and data-parallel GPT-3-style transformer language models built on mesh-tensorflow, with r… | 10 | 8255 | abandoned |
| ChatAnyTeam/ChatAny ChatAny is a self-hostable web application that aggregates ChatGPT-style AI chat with AI image generation services like Midjourney and Stab… | 51 | 6492 | abandoned |
| rustformers/llm An ecosystem of Rust libraries built on GGML for running large language model inference locally, including a CLI binary and reusable crates… | 10 | 6155 | abandoned |
| Lightning-AI/lit-llama Lit-LLaMA is an independent, Apache 2.0-licensed implementation of the LLaMA language model built on nanoGPT, covering pre-training, fine-t… | 40 | 6080 | abandoned |
| qodo-ai/qodo-cover Qodo Cover is an AI-powered tool that automatically generates unit tests to increase code coverage, using LLMs to write and validate tests … | 57 | 5651 | abandoned |
| ramon-victor/freegpt-webui A Python web application providing a ChatGPT-style chat UI backed by the G4F (gpt4free) API, allowing access to GPT-3.5/4 models without an… | 10 | 5611 | abandoned |
| dsdanielpark/Bard-API An unofficial Python package that returns Google Bard (now Gemini) responses by reverse-engineering the web interface using cookie values i… | 10 | 5164 | abandoned |
| opencx-labs/copilot A TypeScript library for building AI copilot/sidekick experiences powered by large language models. The project is no longer maintained, pe… | 10 | 5109 | abandoned |
| AutoGPTQ/AutoGPTQ AutoGPTQ is a Python library for quantizing large language models using the GPTQ weight-only quantization algorithm, with user-friendly API… | 10 | 5065 | abandoned |
| biobootloader/wolverine Wolverine is a Python CLI tool that runs Python scripts and, when they crash, sends the error to GPT-4 which edits the file and reruns it r… | 31 | 5060 | abandoned |
| transcranial/keras-js A JavaScript library for running trained Keras deep learning models directly in the browser, with GPU acceleration via WebGL 2. It also sup… | 23 | 4964 | abandoned |
| Stability-AI/StableSwarmUI StableSwarmUI is a modular web-based user interface for Stable Diffusion image generation, emphasizing accessible powertools, high performa… | 21 | 4947 | abandoned |
| TheoKanning/openai-java A Java client library for the OpenAI API, providing POJOs, a Retrofit-based client, and a convenience service class for GPT-3, ChatGPT, and… | 10 | 4719 | abandoned |
| truefoundry/cognita Cognita is an open-source RAG (Retrieval Augmented Generation) framework by TrueFoundry for building modular, production-ready RAG applicat… | 10 | 4420 | abandoned |
| rawandahmad698/PyChatGPT PyChatGPT is a Python client and CLI for ChatGPT that reverse-engineers the ChatGPT web interface, handling automatic login, access token r… | 70 | 4185 | abandoned |
| nraiden/openv0 openv0 is a generative UI component framework that uses AI to generate and iteratively refine UI components with live preview. It runs a mu… | 28 | 3956 | abandoned |
| karpathy/reader3 A lightweight, self-hosted EPUB reader web app that presents books one chapter at a time so users can easily copy text into an LLM and read… | 37 | 3877 | abandoned |
| WhatsApp ChatGPT AI Assistant A self-hosted WhatsApp bot that connects OpenAI's GPT and DALL-E 2 to WhatsApp, letting users chat with an AI assistant and generate images… | 70 | 3786 | abandoned |
| ravenscroftj/turbopilot TurboPilot is a self-hosted, Copilot-like code completion server that runs large language models (Salesforce CodeGen, StableCode, WizardCod… | 10 | 3772 | abandoned |
| twinnydotdev/twinny Twinny is a free, MIT-licensed Visual Studio Code extension providing AI code completion (fill-in-the-middle) and code chat, working with l… | 94 | 3659 | abandoned |
| mukulpatnaik/researchgpt A FastAPI-based research assistant that lets users have a conversation with any PDF research paper by extracting text, generating embedding… | 10 | 3518 | abandoned |
| gencay/vscode-chatgpt An unofficial Visual Studio Code extension integrating OpenAI ChatGPT, providing sidebar conversations, streaming answers, and code inserti… | 23 | 3448 | abandoned |
| latitudegames/AIDungeon AI Dungeon 2 is an open-source, infinitely generated text adventure game powered by a fine-tuned GPT-2 language model. It can be played loc… | 10 | 3220 | abandoned |
| protectai/llm-guard LLM Guard is a Python library providing input and output scanners to secure LLM interactions, including sanitization, harmful language dete… | 10 | 3208 | abandoned |
| alpa-projects/alpa Alpa is a Python system for training and serving large-scale neural networks by automatically parallelizing single-device code across distr… | 10 | 3176 | abandoned |
| neuralmagic/deepsparse DeepSparse is a sparsity-aware deep learning inference runtime that delivers GPU-class performance on x86 CPUs for ONNX models, covering co… | 10 | 3157 | abandoned |
| Mintlify Documentation Mintlify Writer is an AI-powered editor plugin (VS Code extension and IntelliJ plugin) that generates docstrings and documentation from hig… | 10 | 3127 | abandoned |
| NVIDIA/ChatRTX ChatRTX is a Windows demo application for building personalized RAG chatbots on local RTX GPUs using TensorRT-LLM, NVIDIA NIM, and LlamaInd… | 10 | 3116 | abandoned |
| gragland/chatgpt-chrome-extension A Chrome extension that integrates ChatGPT into every text box on the internet, letting users invoke AI assistance via right-click without … | 32 | 2961 | abandoned |
| janhq/cortex.cpp Cortex.cpp is a C++ local AI platform that runs GGUF (llama.cpp) and ONNX models on your own machine, exposing an OpenAI-compatible HTTP AP… | 10 | 2748 | abandoned |
| npiv/chatblade Chatblade is a Python CLI tool for querying OpenAI's ChatGPT from the terminal, supporting piped input, saved prompt preambles, session rec… | 47 | 2590 | abandoned |
| Azure-Samples/graphrag-accelerator A one-click Azure deployment accelerator that hosts the GraphRAG knowledge-graph-powered RAG system as a scalable API service. It builds on… | 10 | 2406 | abandoned |
| aress31/burpgpt burpgpt is a Burp Suite extension that sends HTTP traffic to OpenAI GPT models for AI-driven passive vulnerability scanning and traffic ana… | 31 | 2394 | abandoned |
| AutoAWQ AutoAWQ is a Python library implementing the Activation-aware Weight Quantization (AWQ) algorithm for 4-bit quantization of large language … | 10 | 2346 | abandoned |
| google-gemini/deprecated-generative-ai-python The deprecated Google AI Python SDK for the Gemini API, which allowed developers to integrate Google's generative AI models into Python app… | 10 | 2326 | abandoned |
| instill-ai/instill-core Instill Core is a full-stack AI infrastructure tool for data, model, and pipeline orchestration, aimed at building AI-first applications wi… | 70 | 2317 | abandoned |
| microsoft/kernel-memory Kernel Memory is a multi-modal AI service and .NET library for indexing datasets through hybrid data pipelines, enabling RAG, semantic sear… | 66 | 2233 | abandoned |
| Aide Aide is an open-source, AI-native code editor built as a fork of VS Code, integrating agentic AI for chat-driven multi-file edits, proactiv… | 10 | 2192 | abandoned |
| run-llama/llama_deploy LlamaDeploy is a Python framework for deploying LlamaIndex agentic workflows to production, with support for multi-agent services. The proj… | 67 | 2068 | abandoned |
| intel/intel-extension-for-pytorch A Python package extending official PyTorch with Intel-specific optimizations for CPUs and GPUs, including quantization and LLM inference a… | 10 | 2012 | abandoned |
| ONNX.js ONNX.js is a JavaScript library for running pre-trained ONNX machine learning models directly in browsers and Node.js, using WebAssembly fo… | 10 | 1815 | abandoned |
| n4ze3m/dialoqbase Dialoqbase is an open-source self-hosted application for creating custom chatbots powered by a personal knowledge base, supporting many cha… | 10 | 1787 | abandoned |
| IntelLabs/fastRAG fastRAG is a Python research framework for building efficient, optimized retrieval-augmented generation (RAG) pipelines, built on Haystack … | 10 | 1786 | abandoned |
| AutoForgeAI/autoforge AutoForge is a long-running autonomous coding agent built on the Claude Agent SDK that builds complete applications across multiple session… | 54 | 1772 | abandoned |
| Onju Voice A hackable AI home assistant platform that replaces the internals of a Google Nest Mini with a custom ESP32-S3 PCB, paired with a server th… | 59 | 1732 | abandoned |
| huggingface/swift-coreml-transformers A Swift library with Core ML implementations of GPT-2, DistilGPT-2, BERT, and DistilBERT, including tokenizers and demo apps for on-device … | 10 | 1683 | abandoned |
| 0x676e67/ninja Ninja is a reverse-engineered ChatGPT proxy server written in Rust that proxies ChatGPT-API/OpenAI-API requests, handles account authentica… | 10 | 1677 | abandoned |
| terrifyzhao/bert-utils A Python library that simplifies Google's BERT to generate sentence vectors with one line of code and perform text classification and text … | 32 | 1669 | abandoned |
| Veloera/Veloera Veloera is a self-hosted AI API gateway written in Go, forked from New API, that proxies and manages access to multiple LLM providers with … | 49 | 1634 | abandoned |
| Binaryify/open-kimi-ppt-skill An unofficial Kimi Slides skill that let AI agents generate editable PPTD and PPTX presentations, bundled with a local browser-based editor… | 10 | 1597 | abandoned |
| MeetKai/functionary Functionary is a chat language model and serving code that interprets and executes functions/plugins, deciding when to call tools in parall… | 66 | 1593 | abandoned |
| ELS-RD/kernl Kernl is an open-source PyTorch model optimization library that accelerates transformer inference on GPUs with a single line of code, using… | 10 | 1583 | abandoned |
| caesarHQ/textSQL textSQL is an open-source project that converts natural language questions into SQL queries using GPT-3.5, letting users query databases wi… | 31 | 1581 | abandoned |
| Farama-Foundation/ChatArena ChatArena is a Python library providing multi-agent language game environments for large language models, built around a Markov Decision Pr… | 34 | 1567 | abandoned |
| SmartSchoolAI/ai-to-pptx Ai-to-pptx is an open-source web application (Vue frontend plus PHP backend) that uses large language models like DeepSeek to generate pres… | 52 | 1460 | abandoned |
| nod-ai/AMD-SHARK-Studio AMD-SHARK Studio is a web UI distribution for high-performance machine learning inference built on SHARK and IREE, primarily for running St… | 45 | 1451 | abandoned |
| idootop/migpt-next MiGPT-Next is a TypeScript library and Docker-runnable service that connects Xiaomi's XiaoAI smart speakers to OpenAI-compatible large lang… | 10 | 1445 | abandoned |
| jamesturk/scrapeghost scrapeghost is an experimental Python library that uses OpenAI's GPT models to scrape structured data from websites without writing page-sp… | 45 | 1439 | abandoned |
| acheong08/Bard A Python SDK and CLI that reverse engineers Google's Bard chatbot API, allowing programmatic access using session cookies. It offers both s… | 10 | 1391 | abandoned |
| prefect-archive/ControlFlow ControlFlow is a Python framework for building agentic AI workflows, letting developers define discrete observable tasks, assign specialize… | 10 | 1387 | abandoned |
| irgolic/AutoPR AutoPR is a GitHub Action that autonomously generated pull requests in response to issues, planning a fix, writing code, pushing a branch, … | 10 | 1373 | abandoned |
| terry3041/pyChatGPT An unofficial Python wrapper that lets scripts interact with ChatGPT by automating a browser session, bypassing Cloudflare anti-bot protect… | 10 | 1335 | abandoned |
| miurla/babyagi-ui BabyAGI UI is a Next.js web application that provides a ChatGPT-like interface for running and developing with the BabyAGI autonomous AI ag… | 10 | 1325 | abandoned |
| CommandCodeAI/BaseAI BaseAI is a TypeScript-first web AI framework for building serverless, composable AI agents with memory (RAG), tools, and pipes, developed … | 56 | 1283 | abandoned |
| ray-project/ray-llm RayLLM was a library for serving and deploying large language models on top of Ray, providing LLM APIs built on Ray Serve. It has been arch… | 10 | 1261 | abandoned |
| OpenRouterArchived/openrouter-runner A deprecated LLM inference engine built with vLLM by OpenRouter in its early days to serve niche fine-tuned models to early adopters. It is… | 10 | 1256 | abandoned |
| Tencent/FeatherCNN FeatherCNN is a high-performance, lightweight CNN inference library developed by Tencent, targeting ARM CPUs on mobile (iOS/Android), embed… | 23 | 1228 | abandoned |
| langchain-ai/langchain-extract A FastAPI-based web server that uses LLMs to extract structured information from text and files, with extractors defined via JSON Schema an… | 10 | 1192 | abandoned |
| kootenpv/neural_complete Neural Complete is a generative LSTM-based autocomplete system trained on Python Keras source code to help write neural network code. It co… | 32 | 1160 | abandoned |
| github/gh-copilot A GitHub CLI extension providing a chat-like AI assistant in the terminal that suggests shell commands and explains what commands do. It ha… | 10 | 1154 | abandoned |
| acheong08/ChatGPT-to-API A Go server that exposes an OpenAI-compatible chat completions API backed by ChatGPT's web interface, cycling through accounts and access t… | 10 | 1138 | abandoned |
| snowby666/poe-api-wrapper A Python API wrapper for Poe.com that provides unofficial access to various AI chatbots like GPT-4, Claude, Llama, Gemini, and Mistral thro… | 10 | 1093 | abandoned |
| google-gemini/deprecated-generative-ai-swift The deprecated Google AI Swift SDK for calling the Gemini API from Apple platform apps. It is explicitly obsolete and developers are direct… | 10 | 1073 | abandoned |
| pinecone-io/canopy Canopy is an open-source Retrieval Augmented Generation (RAG) framework and context engine built on top of the Pinecone vector database, pr… | 10 | 1033 | abandoned |
| graykode/gpt-2-Pytorch A compact PyTorch reimplementation of OpenAI's GPT-2 for text generation, loading Hugging Face's converted pre-trained weights. It provides… | 32 | 1012 | abandoned |
| FutureUniant/WorkShadow WorkShadow is a local-first desktop work journal app built with Tauri, React, and Rust, offering rich-text logging, keyword and semantic se… | 68 | 13 | active |
| tensorflow/tensorflow TensorFlow is an open‑source end‑to‑end platform for building and deploying machine‑learning models. It provides stable Python and C++ APIs… | 83 | 200740 | active |
| Transformers 🤗 Transformers is a Python library that provides a unified API for thousands of pretrained models across text, vision, audio, and multimoda… | 95 | 167798 | active |
| Dify Dify is an open‑source platform for building production‑ready AI agents, RAG pipelines and agentic workflows with a visual no‑code studio. … | 92 | 158944 | active |
| ComfyUI ComfyUI is a modular, node‑based visual interface and inference engine for generative AI. It lets users chain multiple models, adjust every… | 95 | 138077 | active |
| RAGFlow RAGFlow is an open‑source Retrieval‑Augmented Generation engine that combines vector search, BM25 and custom scoring with integrated AI age… | 94 | 92343 | active |
| Odysseus AI Workspace Odysseus is a self-hosted, privacy-first AI workspace that combines chat, autonomous agents, tools/MCP integration, model serving, and deep… | 57 | 87526 | active |
| BettaFish MiroFish is a Python-based swarm intelligence prediction engine that builds high-fidelity parallel digital worlds populated by thousands of… | 67 | 76494 | active |
| career-ops career-ops is an open-source, local-first AI job search system that runs inside AI coding CLIs (Claude Code, Codex, OpenCode, etc.). It sca… | 81 | 73985 | active |
| OpenBB OpenBB is an open-source Python library that provides a unified data integration layer for financial, economic, and crypto data. It powers … | 85 | 73550 | active |
| AnythingLLM AnythingLLM is an all-in-one, local-first AI application for chatting with your documents, running AI agents, and building workflows entire… | 97 | 67412 | active |
| calesthio/OpenMontage OpenMontage is an open-source agentic video production system that turns AI coding assistants (Claude Code, Codex, Cursor, Copilot) into fu… | 59 | 65530 | active |
| upstash/context7 Context7 is a platform that provides up-to-date, version‑specific code documentation and examples directly to LLMs and AI code editors. It … | 81 | 62588 | active |
| sansan0/TrendRadar TrendRadar is a self-hostable AI-powered public opinion and trending-news monitor that aggregates hot topics across multiple platforms and … | 62 | 62570 | active |