Open-source llm-inference software
3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
3358 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| pytorch/glow Glow is a machine learning compiler and execution engine for neural network hardware accelerators, developed under the PyTorch project. It … | 10 | 3316 | maintenance |
| LLM-As-Chatbot A Python application that serves open-source instruction-tuned LLMs as chatbot services via a Gradio web UI or a Discord bot. It integrates… | 31 | 3315 | maintenance |
| X-D-Lab/LangChain-ChatGLM-Webui A Gradio-based web UI that combines LangChain with ChatGLM-6B and other open-source LLMs to answer questions over a local knowledge base. U… | 31 | 3310 | maintenance |
| bytedance/lightseq LightSeq is a high-performance CUDA-based library for training and inference of sequence models like Transformer, BERT, GPT, and BART, with… | 10 | 3292 | maintenance |
| FreeGPT35 A self-hosted service that exposes the login-free ChatGPT Web's GPT-3.5-Turbo as an OpenAI-compatible /v1/chat/completions API. It can be d… | 10 | 3223 | maintenance |
| eureka-research/Eureka Eureka is a research library from NVIDIA that uses coding LLMs like GPT-4 to automatically generate reward functions for reinforcement lear… | 28 | 3210 | maintenance |
| SilasMarvin/lsp-ai LSP-AI is an open-source language server written in Rust that acts as a backend for AI-powered editor features like LLM chat and code compl… | 18 | 3208 | maintenance |
| huawei-noah/Pretrained-Language-Model A collection of pretrained language models and optimization techniques from Huawei Noah's Ark Lab, including PanGu-α (200B-parameter Chines… | 32 | 3166 | maintenance |
| DAMO-NLP-SG/Video-LLaMA Video-LLaMA is an instruction-tuned audio-visual language model that extends LLaMA with video and audio understanding via cross-modal pretr… | 30 | 3140 | maintenance |
| linyiLYi/bilibot A local chatbot fine-tuned from Bilibili user comments, built on Qwen1.5-32B-Chat using Apple's MLX LoRA fine-tuning. It supports text chat… | 25 | 3127 | maintenance |
| Kyle-Ye/XcodeLLMEligible A shell-based utility that overrides macOS eligibility checks to enable Xcode LLM, Apple Intelligence, and iPhone Mirroring on unsupported … | 10 | 3099 | maintenance |
| qwopqwop200/GPTQ-for-LLaMa A Python library that applies GPTQ 4-bit weight quantization to LLaMA large language models, drastically reducing memory usage and checkpoi… | 31 | 3070 | maintenance |
| reloadware/reloadium Reloadium is a Python library and PyCharm plugin providing advanced hot reloading, profiling, and AI-assisted error fixing. It lets develop… | 32 | 2984 | maintenance |
| mpoon/gpt-repository-loader A Python command-line tool that converts a Git repository's files and structure into a single text format suitable for feeding to LLMs. It … | 31 | 2972 | maintenance |
| johnma2006/mamba-minimal A single-file, readable PyTorch implementation of the Mamba selective state space model architecture. It produces numerically equivalent ou… | 27 | 2968 | maintenance |
| FreedomIntelligence/LLMZoo LLM Zoo is a project providing data, models, and evaluation benchmarks for large language models, including the multilingual Phoenix and Ch… | 31 | 2936 | maintenance |
| muellerberndt/mini-agi MiniAGI is a simple general-purpose autonomous AI agent built on the OpenAI API, compatible with GPT-3.5-Turbo and GPT-4. It combines a rob… | 75 | 2932 | maintenance |
| stanford-crfm/helm HELM is a Python framework from Stanford's CRFM for holistic, reproducible evaluation of foundation models including LLMs and multimodal mo… | 87 | 2924 | maintenance |
| Tencent/PhoenixGo PhoenixGo is a Go (weiqi/baduk) AI engine from Tencent that implements the AlphaGo Zero approach of learning the game without human knowled… | 23 | 2918 | maintenance |
| microsoft/genaiscript GenAIScript is a JavaScript/TypeScript framework for programmatically assembling and orchestrating LLM prompts, tools, and data in code. It… | 10 | 2876 | maintenance |
| shreyashankar/gpt3-sandbox A Python library that wraps the OpenAI GPT-3 API to make it easy to prime the model with few-shot examples and launch a local web demo with… | 32 | 2868 | maintenance |
| OpenPipe/OpenPipe OpenPipe is an open-source fine-tuning and model-hosting platform that turns expensive LLM prompts into cheaper fine-tuned models. It offer… | 30 | 2832 | maintenance |
| yanqiangmiffy/Chinese-LangChain A Chinese-language local knowledge base question-answering application built on ChatGLM-6B and LangChain, with a Gradio web UI. It supports… | 30 | 2823 | maintenance |
| microsoftarchive/promptbench PromptBench is a unified Python library from Microsoft for evaluating and understanding large language models across many datasets, models,… | 10 | 2820 | maintenance |
| Alpha-VLLM/LLaMA2-Accessory LLaMA2-Accessory is an open-source Python toolkit for pretraining, finetuning, and deploying large language models and multimodal LLMs, inc… | 29 | 2796 | maintenance |
| henrywoo/pyllama A Python library that is a modified version of Meta's LLaMA implementation, making it easy to download model weights and run LLaMA models o… | 22 | 2778 | maintenance |
| OpenBMB/BMTools BMTools is an open-source Python library that extends large language models with external tools, letting developers build plugins by writin… | 31 | 2773 | maintenance |
| Prompt Engine An NPM utility library from Microsoft for building and maintaining prompts for Large Language Models, with generic, code, and chat prompt e… | 10 | 2757 | maintenance |
| apple/ml-ane-transformers A reference PyTorch implementation of the Transformer architecture optimized for Apple's Neural Engine (ANE), including optimized Hugging F… | 65 | 2745 | maintenance |
| paperswithcode/galai galai is the official Python library for loading and running inference with the GALACTICA general-purpose scientific language models from P… | 23 | 2741 | maintenance |
| ohmplatform/FreedomGPT FreedomGPT is a React and Electron desktop application that runs LLM models locally on Mac, Windows, and Linux using llama.cpp, providing a… | 21 | 2674 | maintenance |
| microsoft/DirectML DirectML is a high-performance, hardware-accelerated DirectX 12 library for machine learning that provides GPU acceleration across DirectX … | 55 | 2595 | maintenance |
| IntelligenzaArtificiale/Free-Auto-GPT A simplified open-source reimplementation of Auto GPT that runs autonomous AI agents without any paid APIs, using free access to models via… | 30 | 2529 | maintenance |
| wenge-research/YAYI YAYI is a family of open-source Chinese-optimized large language models (7B/13B) instruction-tuned from LLaMA 2 and BLOOM base models by We… | 10 | 2524 | maintenance |
| young-geng/EasyLM EasyLM is a JAX/Flax-based framework for pre-training, finetuning, evaluating, and serving large language models like LLaMA. It scales trai… | 32 | 2515 | maintenance |
| AIMedia AIMedia is a heavyweight integrated application that automatically scrapes trending news, generates AI-written articles with AI-generated i… | 66 | 2484 | maintenance |
| Stability-AI/stability-sdk A Python SDK and command-line client for interacting with Stability AI's gRPC-based APIs, primarily for Stable Diffusion image generation a… | 56 | 2435 | maintenance |
| AkariAsai/self-rag Original implementation of Self-RAG, a framework that trains a language model to adaptively retrieve passages on demand and critique its ow… | 28 | 2433 | maintenance |
| zai-org/CogVLM2 CogVLM2 is an open-source multi-modal vision-language model family built on Meta-Llama-3-8B-Instruct, offering image and video understandin… | 25 | 2429 | maintenance |
| microsoft/Codex-CLI A Python CLI tool that uses OpenAI's GPT-3 Codex model to translate natural language commands into Bash, Zsh, or PowerShell commands. Users… | 32 | 2421 | maintenance |
| OpenBMB/CPM-Bee CPM-Bee is a fully open-source, commercially usable 10-billion-parameter bilingual (Chinese-English) foundation large language model traine… | 68 | 2408 | maintenance |
| SoraWebui SoraWebui is an open-source web client for generating videos from text prompts using OpenAI's Sora model, built with TypeScript and deploya… | 17 | 2403 | maintenance |
| IST-DASLab/gptq Reference implementation of GPTQ, a one-shot post-training weight quantization method for large generative transformer models based on appr… | 32 | 2386 | maintenance |
| glowingjade/obsidian-smart-composer Smart Composer is an Obsidian plugin that adds an AI chat assistant with contextual awareness of your vault, smart writing assistance, and … | 58 | 2331 | maintenance |
| TigerResearch/TigerBot TigerBot is a multi-language, multi-task large language model project from TigerResearch, providing pretrained and chat-tuned model weights… | 30 | 2262 | maintenance |
| Turing-Project/AntiFraudChatBot A prompt-based Chinese conversational AI framework that connects a fine-tuned large language model (YUAN-1.0) to WeChat via the wechaty fra… | 23 | 2220 | maintenance |
| ZQCNN ZQCNN is a lightweight deep learning inference framework written in C/C++ that runs on Windows, Linux, and ARM-Linux. It ships with demos f… | 57 | 2212 | maintenance |
| hwchase17/notion-qa A Python application that lets you ask natural-language questions about a Notion database, built on LangChain and OpenAI. It ingests export… | 32 | 2151 | maintenance |
| facebookresearch/chameleon Repository for Meta Chameleon, an early-fusion token-based mixed-modal foundation model that understands and generates interleaved images a… | 10 | 2101 | maintenance |
| camenduru/text-generation-webui-colab A collection of Google Colab notebooks that launch the oobabooga text-generation-webui with a Gradio interface for running large language m… | 31 | 2092 | maintenance |
| ai-forever/ru-gpts A repository of Russian GPT-3 language models (ruGPT3XL/Large/Medium/Small and ruGPT2Large) with usage and fine-tuning examples. It provide… | 32 | 2089 | maintenance |
| sonnylazuardi/chat-ai-desktop An unofficial open-source ChatGPT desktop app for Mac, Windows, and Linux that lives in the menubar, built with Tauri and Rust. It wraps th… | 23 | 1998 | maintenance |
| Fullive-AI/Anima Anima is an open-source Agent OS that gives home hardware perceptive, decision-making, and learning AI capabilities by connecting devices t… | 55 | 639 | active |
| deepseek-ai/DeepSeek-MoE DeepSeekMoE 16B is a Mixture-of-Experts language model with fine-grained expert segmentation and shared expert isolation, released with bas… | 27 | 1981 | maintenance |
| cloudflare/moltworker A Cloudflare Workers project that runs the OpenClaw (formerly Moltbot/Clawdbot) self-hosted personal AI assistant on Cloudflare's Sandbox c… | 49 | 9948 | experimental |
| QwenLM/Qwen-Audio Official repository for Qwen-Audio, Alibaba Cloud's large audio-language model with pretrained and chat variants. It provides model weights… | 28 | 1955 | maintenance |
| liltom-eth/llama2-webui A Gradio-based web UI for running Llama 2 models (7B/13B/70B, GPTQ, GGML, GGUF, CodeLlama) locally on GPU or CPU across Linux, Windows, and… | 20 | 1934 | maintenance |
| nat/natbot natbot is a Python tool that drives a web browser using GPT-3, letting an LLM agent navigate and interact with web pages. It serializes the… | 32 | 1933 | maintenance |
| uTensor uTensor is an extremely lightweight machine learning inference framework built on TensorFlow and optimized for Arm Cortex-M microcontroller… | 28 | 1933 | maintenance |
| S-LoRA/S-LoRA S-LoRA is a serving system for large language models that scales to thousands of concurrent LoRA adapters on one or more GPUs. It uses Unif… | 10 | 1926 | maintenance |
| genspark-ai/genoffice GenOffice is a free, open-source AI office suite for macOS, Windows, and Linux built on Electron, offering a word processor, spreadsheet, p… | 80 | 9695 | experimental |
| OkGoDoIt/OpenAI-API-dotnet An unofficial C#/.NET SDK wrapping the OpenAI API, covering chat completions (GPT-3.5/4), DALL-E image generation, embeddings, moderation, … | 23 | 1894 | maintenance |
| marella/ctransformers CTransformers provides Python bindings for transformer language models implemented in C/C++ using the GGML library. It offers a unified int… | 21 | 1887 | maintenance |
| Forethought-Technologies/AutoChain AutoChain is a lightweight, extensible Python framework for building custom LLM-powered generative agents with custom tools, inspired by La… | 52 | 1882 | maintenance |
| Akegarasu/ChatGLM-webui A Gradio-based web UI for running the THUDM ChatGLM-6B large language model locally. It provides a chat interface with adjustable parameter… | 31 | 1877 | maintenance |
| salesforce/ctrl CTRL is a 1.63 billion-parameter conditional Transformer language model from Salesforce that generates text controllable via control codes … | 10 | 1877 | maintenance |
| clue-ai/ChatYuan ChatYuan is an open-source bilingual (Chinese and English) dialogue large language model in ChatGPT style, with a lightweight variant runna… | 32 | 1860 | maintenance |
| Helixform/CodeCursor CodeCursor is a Visual Studio Code extension that brings the Cursor AI code editor experience into VS Code, offering AI code generation, co… | 10 | 1824 | maintenance |
| openai/gpt-discord-bot An example Discord bot written in Python that uses the OpenAI chat completions API to hold conversations with the gpt-3.5-turbo model and t… | 32 | 1823 | maintenance |
| xmartlabs/Bender Bender is a Swift library that provides an abstraction layer over Apple's MetalPerformanceShaders for defining and running neural networks … | 23 | 1800 | maintenance |
| hieunc229/copilot-clone Captain Stack is a VSCode extension that provides code suggestions similar to GitHub Copilot. Instead of relying solely on AI, it searches … | 32 | 1783 | maintenance |
| ttengwang/Caption-Anything Caption-Anything combines Segment Anything image segmentation, visual captioning, and ChatGPT to generate tailored captions for any object … | 31 | 1777 | maintenance |
| Fictiverse/Redream A Windows desktop application that performs near-realtime Stable Diffusion img2img generation from a captured screen area, using the Automa… | 23 | 1776 | maintenance |
| OpenCodeInterpreter/OpenCodeInterpreter OpenCodeInterpreter is a suite of open-source code generation models and deployment code that integrates code execution and iterative refin… | 26 | 1770 | maintenance |
| flowtyone/flowty-realtime-lcm-canvas A real-time sketch-to-image demo application that uses Latent Consistency Models (LCM) with a Gradio web UI. Users draw on a canvas and see… | 27 | 1768 | maintenance |
| Synaptrix/ChatGPT-Desktop A cross-platform desktop ChatGPT client built with Tauri, Vue 3, and Arco Design that wraps the OpenAI GPT-3.5-turbo API. It offers a fast … | 10 | 1767 | maintenance |
| nashsu/FreeAskInternet FreeAskInternet is a self-hosted, Perplexity-like search aggregator and answer generator that runs multi-engine searches via SearXNG and fe… | 25 | 8749 | experimental |
| TypeChat TypeChat is a TypeScript library from Microsoft that replaces prompt engineering with 'schema engineering' for building natural language in… | 75 | 8692 | experimental |
| pydantic/monty Monty is a minimal, secure Python interpreter written in Rust designed for safely executing LLM-generated code without container-based sand… | 96 | 8621 | experimental |
| SecureAI-Tools/SecureAI-Tools A self-hosted web application for private AI productivity, offering chat with AI models and chat-with-documents (PDF) backed by local infer… | 18 | 1727 | maintenance |
| 101dotxyz/GPTeam GPTeam is an open-source multi-agent simulation that uses GPT-4 to create multiple AI agents with memory who collaborate and communicate to… | 74 | 1724 | maintenance |
| cvlab-columbia/viper ViperGPT is a research codebase that composes vision-and-language models with code generated by large language models (GPT-3.5/GPT-4) to pe… | 31 | 1720 | maintenance |
| Maratyszcza/NNPACK NNPACK is a C99 acceleration package providing high-performance SIMD and multi-core CPU implementations of neural network layers, especiall… | 32 | 1710 | maintenance |
| juanjuandog/FinSight-AI FinSight AI is an open-source equity research workspace for A-share companies that turns market data, filings, and financial metrics into s… | 56 | 559 | active |
| BotGroup.Chat A multi-user AI group chat application built with React and deployable to Cloudflare Pages, where multiple AI characters with distinct pers… | 67 | 1702 | maintenance |
| techleadhd/chatgpt-retrieval A simple Python script that lets you ask ChatGPT questions about your own files using LangChain and a Chroma vector store. It loads text or… | 30 | 1697 | maintenance |
| ELS-RD/transformer-deploy A Python tool that optimizes Hugging Face transformer models for production inference using ONNX Runtime and NVIDIA TensorRT, deploying the… | 23 | 1695 | maintenance |
| zlwaterfield/scramble Scramble is an open-source browser extension that uses LLMs to enhance writing directly on any webpage, acting as a customizable Grammarly … | 31 | 1688 | maintenance |
| TheExplainthis/ChatGPT-Line-Bot A Python application that integrates ChatGPT into the LINE messaging platform, letting users chat with GPT models and generate DALL·E 2 ima… | 22 | 1687 | maintenance |
| mit-han-lab/smoothquant SmoothQuant is a training-free post-training quantization (PTQ) method that enables W8A8 (8-bit weight, 8-bit activation) quantization for … | 32 | 1686 | maintenance |
| eyurtsev/kor Kor is a Python library for extracting structured data from text using large language models. Users define a schema (Kor-style or Pydantic)… | 22 | 1684 | maintenance |
| hamelsmu/evals-skills A collection of skills (prompts/instructions) that guide AI coding agents like Claude Code through building and auditing LLM evaluation pip… | 10 | 1677 | maintenance |
| lxfater/Portal Portal is a cross-platform desktop tool that integrates ChatGPT capabilities into any application via a global keyboard shortcut. It sends … | 10 | 1673 | maintenance |
| LangGraph A Python library for building hierarchical multi-agent systems on top of LangGraph, where a central supervisor agent orchestrates and deleg… | 10 | 1656 | maintenance |
| TrainPPTAgent MultiAgentPPT is a Python-based multi-agent system that generates editable presentation slides from a topic using A2A, MCP, and Google ADK … | 64 | 1643 | maintenance |
| altryne/chatGPT-telegram-bot A Telegram bot that lets users chat with ChatGPT, originally by driving the ChatGPT web interface in Chromium via Playwright, with plans to… | 23 | 1637 | maintenance |
| mmz-001/knowledge_gpt KnowledgeGPT is a Streamlit web application that lets users upload documents (primarily PDFs) and ask questions about them, receiving answe… | 10 | 1628 | maintenance |
| CodeShell CodeShell is a family of 7-billion-parameter code large language models developed by PKU-KCL, trained on 500 billion tokens with an 8192-to… | 28 | 1618 | maintenance |
| wangzhaode/mnn-llm mnn-llm is a C++ library for deploying large language models on-device using Alibaba's MNN inference engine, with demos for CLI, web, Andro… | 21 | 1615 | maintenance |
| divamgupta/stable-diffusion-tensorflow A TensorFlow/Keras implementation of Stable Diffusion with weights ported from the original implementation. It provides a Python API and Co… | 32 | 1608 | maintenance |