Open-source llm-inference software
3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
3358 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| sobelio/llm-chain llm-chain is a Rust crate for building chains of large language model calls to complete complex tasks such as text summarization. It provid… | 21 | 1602 | maintenance |
| RomanHotsiy/commitgpt commitgpt is a Node.js CLI tool that automatically generates commit messages using ChatGPT. It runs `git diff --cached`, sends the diff to … | 32 | 1590 | maintenance |
| jina-ai/thinkgpt ThinkGPT is a Python library implementing Chain of Thought techniques for LLMs, providing memory, self-refinement, knowledge compression, a… | 30 | 1583 | maintenance |
| finic-ai/rag-stack RAGstack is a self-hostable private ChatGPT alternative that deploys open-source LLMs (Llama 2, Falcon, GPT4All) alongside the Qdrant vecto… | 74 | 1581 | maintenance |
| TsinghuaAI/CPM-1-Generate CPM-Generate provides inference code for CPM-LM, a 2.6B-parameter Chinese pre-trained language model based on the GPT-2 architecture and Me… | 32 | 1579 | maintenance |
| microsoft/TinyTroupe TinyTroupe is an experimental Python library for simulating people with specific personalities, interests, and goals using LLM-powered agen… | 74 | 7612 | experimental |
| Frrrrrrrrank/auto_job__find__chatgpt__rpa A free Python RPA script that uses OpenAI's API to automatically generate cover letters from your resume and job descriptions, then sends m… | 27 | 1571 | maintenance |
| Tencent/TurboTransformers A fast, user-friendly C++/Python runtime for transformer model inference (BERT, ALBERT, RoBERTa, GPT2, decoders) on CPU and GPU. It works a… | 34 | 1551 | maintenance |
| pytorch/QNNPACK QNNPACK is a mobile-optimized C library of high-performance kernels for 8-bit quantized neural network operators such as convolution, pooli… | 10 | 1551 | maintenance |
| GoogleCloudPlatform/localllm A CLI tool and Docker image for running quantized LLMs locally on Google Cloud Workstations using llama-cpp-python's web server. It package… | 10 | 1550 | maintenance |
| Stability-AI/sd3.5 A minimal inference-only reference implementation of Stable Diffusion 3.5 and SD3, including text encoders, VAE decoder, MM-DiT core, and S… | 23 | 1536 | maintenance |
| bes-dev/stable_diffusion.openvino A Python CLI implementation of Stable Diffusion text-to-image generation optimized for Intel CPUs and GPUs via OpenVINO. It supports text-t… | 32 | 1534 | maintenance |
| Tameyer41/liftoff Liftoff is a web application that simulates technical mock interviews and provides AI-powered feedback using OpenAI Whisper for speech tran… | 30 | 1524 | maintenance |
| alexrozanski/LlamaChat LlamaChat is a native macOS app for chatting with LLaMA, Alpaca, GPT4All and similar models running entirely locally on your Mac. It is bui… | 57 | 1508 | maintenance |
| run-llama/llama-lab Llama Lab is a collection of experimental projects built on top of LlamaIndex for exploring cutting-edge LLM agent capabilities. It include… | 30 | 1508 | maintenance |
| OWD-AI/AutoAgents AutoAgents is a Python framework that uses an LLM to automatically generate multiple specialized agent roles that collaborate to accomplish… | 44 | 1495 | maintenance |
| rotemweiss57/gpt-newspaper GPT Newspaper is a Python application built on LangGraph that runs a multi-agent pipeline to search, curate, write, critique, design, and p… | 26 | 1472 | maintenance |
| joshpxyne/gpt-migrate GPT-Migrate is a Python CLI tool that uses LLMs (GPT-4 via OpenAI or OpenRouter) to automatically migrate codebases from one framework or l… | 29 | 6972 | experimental |
| CStanKonrad/long_llama LongLLaMA is a large language model handling long contexts of 256k tokens or more, built on OpenLLaMA and fine-tuned with the Focused Trans… | 29 | 1461 | maintenance |
| mckaywrigley/clarity-ai Clarity AI is an open-source clone of Perplexity.ai that answers user queries by scraping Google search results, parsing the webpages, and … | 31 | 1423 | maintenance |
| vas3k/TaxHacker TaxHacker is a self-hosted AI-powered accounting app for freelancers, indie-hackers, and small businesses. It uses LLMs to OCR and analyze … | 82 | 6729 | experimental |
| InstructLab InstructLab (ilab) is an open-source CLI and Python package for enhancing large language models through community-contributed taxonomy-base… | 10 | 1418 | maintenance |
| ConnorJL/GPT2 A community Python/TensorFlow implementation of GPT-2 model training and text generation that supports both GPUs and TPUs. It includes scri… | 32 | 1414 | maintenance |
| a16z-infra/llama2-chatbot A Streamlit-based chatbot web app for chatting with LLaMA 2 models (7B, 13B, 70B) hosted on Replicate API endpoints. It includes per-sessio… | 29 | 1409 | maintenance |
| safety-research/bloom Bloom is a Python tool that automatically generates behavioral evaluation suites for LLMs, probing target models for behaviors like sycopha… | 52 | 1408 | maintenance |
| nraiden/cofounder Cofounder is an AI-powered framework that generates full-stack web applications, including backend, database, and stateful frontend with ge… | 23 | 6653 | experimental |
| ChineseAiDungeon A Chinese-language clone of AI Dungeon that uses OpenAI's ChatGPT API as the storytelling model. It offers both a CLI and a simple GUI app … | 23 | 1404 | maintenance |
| neo4j/NaLLM NaLLM is a demo application from Neo4j exploring synergies between Neo4j graph databases and Large Language Models. It provides a backend A… | 10 | 1404 | maintenance |
| Yifan-Song793/RestGPT RestGPT is an LLM-based autonomous agent that controls real-world applications like TMDB and Spotify by planning and executing RESTful API … | 29 | 1403 | maintenance |
| gotzmann/llama.go A pure-Go reimplementation of llama.cpp-style LLM inference, running LLaMA-family models on CPU without C++ dependencies. It includes tenso… | 21 | 1402 | maintenance |
| 198808xc/Pangu-Weather Official implementation of Pangu-Weather, a deep learning model for fast, accurate medium-range global weather forecasting using 3D neural … | 22 | 1398 | maintenance |
| doonny/PipeCNN PipeCNN is an OpenCL-based FPGA accelerator for large-scale convolutional neural network inference, written in C with pipelined kernels. It… | 32 | 1393 | maintenance |
| anus-dev/ANUS ANUS is a Grok-powered AI agent for the terminal, distributed as an npm CLI, whose stated goal is to evolve into an agent that contributes … | 86 | 6549 | experimental |
| nishiwen1214/ChatReviewer ChatReviewer is a Python application built on the ChatGPT API that analyzes academic papers, summarizing their strengths and weaknesses and… | 31 | 1377 | maintenance |
| keirp/automatic_prompt_engineer Automatic Prompt Engineer (APE) is a Python library implementing the research method from 'Large Language Models Are Human-Level Prompt Eng… | 32 | 1367 | maintenance |
| strnad/CrewAI-Studio CrewAI Studio is a Streamlit-based graphical application for creating, managing, and running CrewAI multi-agent crews and tasks without wri… | 83 | 1360 | maintenance |
| Jackywine/Bella Bella is a self-hosted Node.js web application that acts as a personalized AI digital companion with voice interaction. It combines Whisper… | 45 | 6359 | experimental |
| yuvalsuede/ai-component-generator A Next.js web application that generates UI components (HTML, Tailwind CSS, React, Material UI) from free-form text prompts using OpenAI's … | 31 | 1344 | maintenance |
| chengzeyi/stable-fast Stable Fast is an ultra lightweight performance optimization framework for HuggingFace Diffusers pipelines on NVIDIA GPUs. It achieves SOTA… | 21 | 1305 | maintenance |
| ItsPi3141/alpaca-electron Alpaca Electron is an Electron desktop app that provides a simple ChatGPT-style UI for chatting with Alpaca and other LLaMA-based local LLM… | 21 | 1302 | maintenance |
| apache/maka Apache Maka (Incubating) is a local-first AI agent workspace that runs agents on your machine through a single Runtime Host, executing tool… | 78 | 5861 | experimental |
| goldfishh/chatgpt-tool-hub An open-source Python tool engine that lets ChatGPT (and other LLMs) invoke tools like web search, math computation, code execution, and co… | 21 | 1264 | maintenance |
| UKPLab/EasyNMT EasyNMT is a Python library providing easy access to state-of-the-art neural machine translation for 100+ languages using models like Opus-… | 23 | 1261 | maintenance |
| andrewyng/translation-agent A Python demonstration library by Andrew Ng implementing an agentic machine translation workflow: an LLM translates text, reflects on its o… | 25 | 5816 | experimental |
| ShannonAI/service-streamer Service Streamer is a Python middleware that queues discrete web service requests into mini-batches for deep learning model inference, impr… | 32 | 1241 | maintenance |
| mozilla-ai/any-agent any-agent is a Python library providing a single unified interface for building, serving, and evaluating AI agents across multiple agent fr… | 69 | 1236 | maintenance |
| google-gemini/deprecated-generative-ai-js The deprecated Google AI JavaScript/TypeScript SDK for the Gemini API, now in limited maintenance with support ending November 30, 2025. Us… | 10 | 1234 | maintenance |
| srush/MiniChain MiniChain is a tiny Python library for building applications with large language models by annotating Python functions that call LLMs and c… | 22 | 1232 | maintenance |
| UMass-Embodied-AGI/3D-LLM 3D-LLM is the research code for a large language model that takes 3D representations (objects and scenes) as input, built on BLIP-2/LAVIS. … | 29 | 1217 | maintenance |
| nihui/realsr-ncnn-vulkan A command-line tool implementing the RealSR real-world super-resolution model using the ncnn inference framework with Vulkan GPU accelerati… | 23 | 1217 | maintenance |
| m1guelpf/auto-commit A Rust CLI tool that uses OpenAI's GPT-3.5 to automatically generate commit messages from your staged git changes. It supports dry-run outp… | 23 | 1213 | maintenance |
| dotnet/modernize-dotnet An AI-powered GitHub Copilot agent plugin that analyzes .NET applications, generates upgrade plans, and applies code changes to modernize t… | 73 | 1206 | maintenance |
| RKNN RKNN-Toolkit2 is Rockchip's software development kit for converting trained AI models to RKNN format and deploying them on Rockchip NPU chi… | 23 | 1203 | maintenance |
| KwaiKEG/KwaiAgents KwaiAgents is an open-source suite from Kuaishou for building generalized information-seeking agents powered by LLMs. It includes KAgentSys… | 27 | 1201 | maintenance |
| LiuHC0428/LAW-GPT A Chinese-language legal conversational model (LawGPT_zh / 獬豸) fine-tuned from ChatGLM-6B with LoRA on legal Q&A datasets, including answer… | 31 | 1194 | maintenance |
| IndieKKY/bilibili-subtitle A browser extension (哔哔君) that displays a clickable subtitle list panel for Bilibili videos, enabling jump-to-timestamp navigation, subtitl… | 49 | 1183 | maintenance |
| visual-openllm/visual-openllm An open-source tool that interactively connects different visual models with an LLM, built on ChatGLM, Visual ChatGPT, and Stable Diffusion… | 31 | 1183 | maintenance |
| VoltaML/voltaML VoltaML is a lightweight Python library that compiles and optimizes machine learning and deep learning models for high-performance inferenc… | 32 | 1177 | maintenance |
| punica-ai/punica Punica is a Python system for serving many LoRA-finetuned LLMs from a single copy of the base model on one GPU, using a custom CUDA kernel … | 10 | 1175 | maintenance |
| bestony/ChatGPT-Feishu A ChatGPT bot for Feishu (Lark) that lets users chat with OpenAI's models directly in Feishu messages. It is deployed as a single Node.js e… | 22 | 1171 | maintenance |
| abhagsain/ai-cli A GPT-3/ChatGPT-powered command-line tool that answers CLI command questions directly from the terminal. Built with oclif in TypeScript and… | 23 | 1169 | maintenance |
| uber-research/PPLM PPLM (Plug and Play Language Model) is a research implementation for controlled text generation that steers the topic and attributes of GPT… | 32 | 1152 | maintenance |
| sentient-agi/ROMA ROMA (Recursive Open Meta-Agent) is a Python meta-agent framework for building hierarchical, high-performance multi-agent systems. It enabl… | 46 | 5179 | experimental |
| lupantech/chameleon-llm Chameleon is a research framework for plug-and-play compositional reasoning with large language models, where an LLM planner synthesizes pr… | 21 | 1143 | maintenance |
| peterw/Chat-with-Github-Repo A Python application that lets you chat with any GitHub repository. It clones a repo, embeds its documents into Activeloop Deep Lake, and s… | 30 | 1142 | maintenance |
| Open Interpreter 01 An open-source voice interface platform that lets users control computers conversationally, powered by Open Interpreter. It pairs a Python … | 17 | 5158 | experimental |
| fixie-ai/ai-jsx AI.JSX is a JavaScript/TypeScript framework for building AI applications using JSX components, supporting prompt engineering, tools, docume… | 30 | 1134 | maintenance |
| google-deepmind/dramatron Dramatron is a research tool from DeepMind that uses pre-trained large language models to hierarchically co-write theatre scripts and scree… | 32 | 1133 | maintenance |
| microsoft/Windows-Machine-Learning Microsoft's Windows Machine Learning samples and tools repository, providing a high-performance ONNX inference API powered by ONNX Runtime … | 35 | 1124 | maintenance |
| LLMPerf LLMPerf is a Python library for benchmarking and validating the performance of LLM APIs. It runs load tests measuring inter-token latency a… | 10 | 1123 | maintenance |
| openTrinity/mycontext MyContext is a local-first desktop application that builds a persistent personal work context layer by connecting signals from instant mess… | 55 | 5018 | experimental |
| PaddlePaddle/Paddle.js Paddle.js is a browser-based deep learning inference engine for Baidu PaddlePaddle models, running via WebGL, WebGPU, or WebAssembly backen… | 23 | 1104 | maintenance |
| rlancemartin/auto-evaluator A lightweight Streamlit-based evaluation tool for LLM question-answering chains built with LangChain. It auto-generates question-answer pai… | 30 | 1103 | maintenance |
| DonTizi/rlama RLAMA is a Go CLI tool for building and querying local Retrieval-Augmented Generation (RAG) systems over your documents using Ollama models… | 35 | 1096 | maintenance |
| replit/ReplitLM Official repository with inference code, configs, and guides for the ReplitLM family of code-oriented language models, such as replit-code-… | 10 | 1093 | maintenance |
| WangRongsheng/XrayGLM XrayGLM is the first Chinese multimodal medical large language model that generates radiology report summaries from chest X-ray images, bui… | 30 | 1086 | maintenance |
| openai/automated-interpretability OpenAI's code and tools for automatically generating, simulating, and scoring explanations of neuron behavior in language models, based on … | 10 | 1080 | maintenance |
| Stable Diffusion NCNN A C++ implementation of Stable Diffusion using the NCNN inference framework, supporting both txt2img and img2img. It runs on x86 Windows ex… | 23 | 1067 | maintenance |
| ThePrimeagen/99 A Neovim plugin providing an agentic AI coding experience, integrating LLM providers like OpenCode and Claude Code into the editor. It augm… | 52 | 4743 | experimental |
| OpenBMB/VisCPM VisCPM is a family of open-source bilingual (Chinese/English) multimodal large models built on the 10B CPM-Bee language model, comprising V… | 29 | 1062 | maintenance |
| obiscr/ChatGPT A JetBrains IDE plugin that integrates ChatGPT into IntelliJ-based IDEs, providing an in-editor chat interface with OpenAI's language model… | 32 | 1059 | maintenance |
| mpaepper/llm_agents A small Python library for building agents controlled by large language models, inspired by LangChain but implemented from scratch in very … | 40 | 1054 | maintenance |
| huggingface/optimum-quanto Optimum Quanto is a PyTorch quantization backend for Hugging Face Optimum that quantizes model weights (int2/int4/int8/float8) and activati… | 66 | 1053 | maintenance |
| CSHaitao/LexiLaw LexiLaw is a fine-tuned Chinese legal large language model based on ChatGLM-6B, providing legal consultation and Q&A capabilities. It inclu… | 58 | 1046 | maintenance |
| dandelionsllm/pandallm Panda is an open-source project for overseas Chinese large language models, providing PandaLLM model weights (continued pretraining of LLaM… | 30 | 1027 | maintenance |
| microsoft/Llama-2-Onnx Microsoft's optimized ONNX export of Meta's Llama 2 models (7B and 13B, pretrained and fine-tuned, float16/float32), distributed via Git su… | 29 | 1025 | maintenance |
| CaviraOSS/OpenMemory OpenMemory is a local-first, self-hosted cognitive memory engine that gives LLM applications and AI agents persistent long-term memory, wit… | 66 | 4523 | experimental |
| turtlesoupy/this-word-does-not-exist A project that trains a GPT-2 variant to invent fake English words with generated definitions and example sentences, powering the thiswordd… | 68 | 1024 | maintenance |
| zabirauf/AutoGPT.js AutoGPT.js is an open-source web application that runs AutoGPT-style autonomous GPT agents directly in the browser. It supports file creati… | 31 | 1024 | maintenance |
| awslabs/multi-model-server Multi Model Server (MMS) is a tool for serving deep learning model inference over HTTP endpoints, supporting models from any ML/DL framewor… | 10 | 1024 | maintenance |
| cbh123/narrator A Python app that watches your webcam and generates David Attenborough-style narration of what it sees, using GPT vision models and ElevenL… | 66 | 4428 | experimental |
| varunshenoy/GraphGPT GraphGPT is a web application that converts unstructured natural language text into a knowledge graph using GPT-3, visualizing entities and… | 32 | 4423 | experimental |
| aws-samples/bedrock-access-gateway A proxy service that exposes Amazon Bedrock foundation models through OpenAI-compatible RESTful APIs, letting existing OpenAI SDK-based app… | 10 | 1004 | maintenance |
| fcakyon/autollm AutoLLM is a Python library for rapidly building RAG-based LLM web apps and APIs, offering a unified API over 100+ LLM providers and 20+ ve… | 10 | 1003 | maintenance |
| Plug-and-Play Diffusion Features Official PyTorch implementation of Plug-and-Play (PnP) diffusion features for text-driven image-to-image translation, presented at CVPR 202… | 31 | 1001 | maintenance |
| susiai/susi_device SUSI Device provides sources to install the SUSI AI assistant stack on a Raspberry Pi, combining microphone/speaker, a small display, a loc… | 32 | 1000 | maintenance |
| aiwaves-cn/RecurrentGPT RecurrentGPT is a Python implementation of a paper-proposed framework that simulates LSTM-style recurrence using natural language and promp… | 30 | 999 | maintenance |
| build-with-groq/g1 g1 is a Python prototype that uses prompting strategies with Llama-3.1 70b on Groq to produce o1-like visible reasoning chains for solving … | 46 | 4162 | experimental |
| keijiro/AICommand A proof-of-concept Unity Editor extension that integrates ChatGPT, letting users control the Unity Editor via natural language prompts. It … | 31 | 4095 | experimental |
| Significant-Gravitas/Auto-GPT-Plugins A collection of official and community plugins for Auto-GPT, the autonomous AI agent framework, providing additional capabilities like web … | 10 | 3821 | experimental |