Open-source llm-inference software
3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
3358 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| zgsm-ai/costrict CoStrict is an open-source, enterprise-focused AI coding assistant delivered as VS Code and JetBrains extensions, offering strict-mode code… | 81 | 4444 | active |
| ibttf/interview-coder Interview Coder is an Electron-based desktop application that provides AI assistance during technical interviews, using GPT/OpenAI models t… | 40 | 4444 | active |
| zai-org/GLM-4.5 GLM-4.5 (and successors GLM-4.6/4.7) is a family of open-weight Mixture-of-Experts foundation models from Z.ai focused on agentic tasks, re… | 44 | 4424 | active |
| modelscope/ms-agent MS-Agent is a lightweight Python framework from ModelScope for building LLM-powered agents with autonomous exploration capabilities. It sup… | 86 | 4416 | active |
| open-compass/VLMEvalKit VLMEvalKit is an open-source Python toolkit for evaluating large vision-language models (LMMs/LVLMs) across 80+ benchmarks with support for… | 61 | 4412 | active |
| meta-llama/PurpleLlama Purple Llama is Meta's umbrella project of tools and benchmarks for assessing and improving the security of large language models. It inclu… | 68 | 4401 | active |
| Kiro Kiro is an agentic IDE and CLI that helps developers go from prototype to production using spec-driven development, agent hooks, powers, an… | 64 | 4372 | active |
| Doriandarko/maestro Maestro is a Python framework that uses an orchestrator LLM (originally Claude Opus) to break down objectives into sub-tasks, delegate them… | 26 | 4357 | active |
| BuilderIO/micro-agent Micro Agent is an open-source CLI AI agent that generates unit tests from a natural language prompt and then iterates on code until all tes… | 25 | 4335 | active |
| whoiskatrin/sql-translator SQL Translator is a free, open-source web application that converts natural language queries into SQL code and translates SQL back into hum… | 41 | 4328 | active |
| UditAkhourii/adhd ADHD is a TypeScript skill for coding agents built on the Claude and Codex Agent SDKs that implements parallel divergent ideation via tree-… | 61 | 4326 | active |
| ymcui/Chinese-LLaMA-Alpaca A project releasing Chinese-adapted LLaMA and instruction-tuned Alpaca large language models with extended Chinese vocabulary, plus pre-tra… | 53 | 18936 | maintenance |
| ModelTC/LightLLM LightLLM is a Python-based LLM inference and serving framework designed for lightweight deployment, easy scalability, and high throughput. … | 82 | 4317 | active |
| tloen/alpaca-lora A repository of scripts for reproducing Stanford Alpaca-style instruction tuning of LLaMA models using low-rank adaptation (LoRA) on consum… | 31 | 18899 | maintenance |
| algorithmicsuperintelligence/optillm OptiLLM is an OpenAI API-compatible optimizing inference proxy that applies 20+ inference-time techniques (e.g., Mixture of Agents, MCTS, c… | 85 | 4307 | active |
| leetcode-mafia/cheetah Cheetah is a macOS app that provides real-time AI coaching for software engineering interviews. It transcribes conversation audio locally w… | 21 | 4260 | active |
| jwohlwend/boltz Boltz is a family of open-source biomolecular interaction models (Boltz-1 and Boltz-2) that predict complex structures and binding affiniti… | 58 | 4257 | active |
| ArcInstitute/evo2 Evo 2 is a DNA foundation language model (1B-40B parameters) that models genomes at single-nucleotide resolution with up to 1 million base … | 56 | 4247 | active |
| danveloper/flash-moe An application for running large language models locally on small laptops, written in Objective-C. It enables on-device inference of big mo… | 44 | 4238 | active |
| nyldn/claude-octopus A Claude Code plugin that orchestrates multiple AI model providers (Codex, Copilot, Gemini, Ollama, and others) against the same task, usin… | 79 | 4224 | active |
| zhayujie/bot-on-anything Bot on Anything is a lightweight Python framework for building AI chatbots that connect large language models (ChatGPT, Claude, Gemini, ERN… | 59 | 4222 | active |
| eth-sri/lmql LMQL is a query/programming language for large language models, implemented as a Python-based superset that lets developers interleave prom… | 29 | 4221 | active |
| SylphAI-Inc/AdalFlow AdalFlow is a PyTorch-like Python library for building and auto-optimizing LLM application workflows such as chatbots, RAG pipelines, and a… | 62 | 4220 | active |
| sooryathejas/METATRON METATRON is a CLI-based AI penetration testing assistant that runs a local LLM (via Ollama) entirely offline on Linux, primarily Parrot OS.… | 46 | 4178 | active |
| dataelement/Clawith Clawith is an open-source multi-agent collaboration platform where each AI agent has a persistent identity, long-term memory, and its own w… | 76 | 4164 | active |
| Spark NLP Spark NLP is an open-source natural language processing library built natively on Apache Spark, providing scalable NLP annotations and tran… | 98 | 4160 | stable |
| city96/ComfyUI-GGUF A ComfyUI custom node pack that adds GGUF quantization support for loading diffusion models like FLUX and Stable Diffusion 3.5 in low-bitra… | 48 | 4138 | active |
| zml/zml ZML is a production LLM inference stack written in Zig, built on MLIR and OpenXLA, that compiles models to run at peak performance across N… | 69 | 4136 | active |
| IDEA-CCNL/Fengshenbang-LM Fengshenbang-LM is an open-source suite of Chinese large language models and a PyTorch training framework from IDEA Research's CCNL lab, ai… | 67 | 4123 | active |
| openai/transformer-debugger Transformer Debugger (TDB) is an OpenAI tool for investigating specific behaviors of small language models, combining automated interpretab… | 57 | 4123 | active |
| Langroid Langroid is a Python framework for building LLM-powered applications using multi-agent programming, where agents encapsulate LLM conversati… | 94 | 4120 | active |
| ucbepic/docetl DocETL is a Python library and CLI for building LLM-powered data processing and ETL pipelines over structured and unstructured data using d… | 78 | 4114 | active |
| CommandCodeAI/command-code Command Code is a terminal-based AI coding agent that builds full-stack projects, fixes bugs, writes tests, and refactors code. Its disting… | 66 | 4112 | active |
| copilot.lua A pure Lua Neovim plugin that replaces copilot.vim, providing GitHub Copilot AI code suggestions with a Lua API for integration. It include… | 99 | 4098 | active |
| rahulnyk/knowledge_graph A Python/Jupyter Notebook project that converts any text corpus into a knowledge graph by extracting concepts with an LLM and linking co-oc… | 68 | 4087 | active |
| TransformerOptimus/SuperAGI SuperAGI is a dev-first open source framework for building, managing, and running autonomous AI agents powered by large language models. It… | 21 | 17702 | maintenance |
| FedML FedML (TensorOpera) is a unified Python library for large-scale distributed training, model serving, and federated learning across GPU clou… | 41 | 4070 | active |
| hanshuaikang/AI-Media2Doc A self-hostable web application that uses AI large language models to convert video and audio into various document styles such as Xiaohong… | 52 | 4052 | active |
| anymorph-ai/Claudable Claudable is an open-source, Next.js-based web app builder that uses local CLI coding agents like Claude Code, Codex, Gemini CLI, Qwen Code… | 39 | 4052 | active |
| defog-ai/sqlcoder SQLCoder is a family of fine-tuned open-weight LLMs that convert natural language questions into SQL queries, outperforming GPT-4 on text-t… | 29 | 4048 | active |
| kagent-dev/kagent kagent is a Kubernetes-native framework for building, deploying, and managing AI agents as first-class Kubernetes resources via CRDs. It su… | 84 | 4047 | active |
| googleapis/python-genai The official Google Gen AI Python SDK for integrating Google's generative models (Gemini Developer API and Gemini Enterprise Agent Platform… | 85 | 4035 | active |
| steipete/oracle Oracle is a TypeScript CLI and MCP server that bundles a prompt with selected project files and sends that context to frontier AI models (G… | 82 | 4030 | active |
| QwenLM/Qwen3-Omni Qwen3-Omni is a natively end-to-end omni-modal large language model from Alibaba Cloud's Qwen team that understands text, images, audio, an… | 49 | 4028 | active |
| knownsec/aipyapp AiPy (Python-Use) is an AI desktop assistant from Knownsec that implements a 'Code is Agent' paradigm: it gives an LLM a full Python execut… | 57 | 4027 | active |
| microsoft/PromptWizard PromptWizard is a discrete prompt optimization framework from Microsoft that uses a self-evolving mechanism where an LLM generates, critiqu… | 42 | 4010 | active |
| DaKheera47/job-ops A self-hosted job search assistant that aggregates searches across 10+ job boards, AI-scores role fit, tailors CVs per posting, and tracks … | 82 | 4004 | active |
| badrisnarayanan/antigravity-claude-proxy A Node.js proxy server that exposes Antigravity's Cloud Code Claude and Gemini models behind an Anthropic-compatible Messages API. It trans… | 74 | 4004 | active |
| Rapid-MLX Rapid-MLX is a local LLM inference engine for Apple Silicon Macs built on MLX, exposing a drop-in OpenAI/Anthropic-compatible API. It offer… | 79 | 4002 | active |
| gusye1234/nano-graphrag nano-graphrag is a lightweight, ~1100-line Python implementation of Microsoft's GraphRAG, designed to be small, fast, and easy to read or h… | 40 | 4001 | active |
| urchade/GLiNER GLiNER is a Python framework for training and deploying lightweight, zero-shot Named Entity Recognition models that can extract arbitrary e… | 87 | 3997 | active |
| jackMort/ChatGPT.nvim A Neovim plugin written in Lua that integrates OpenAI's ChatGPT API directly into the editor. It provides interactive Q&A, persona-based co… | 47 | 3992 | active |
| LLaDA Official PyTorch implementation of LLaDA, a family of large language diffusion models (8B base/instruct, MoE, and iLLaDA variants) with pre… | 59 | 3986 | active |
| IREE IREE is an MLIR-based end-to-end machine learning compiler and runtime that lowers models from frameworks like PyTorch, TensorFlow, JAX, an… | 95 | 3980 | active |
| nicobailon/pi-subagents A Pi extension that lets the Pi coding agent delegate tasks to focused child subagents with async execution, truncation, artifacts, and ses… | 80 | 3967 | active |
| circlemind-ai/fast-graphrag fast-graphrag is a Python library providing a streamlined, promptable GraphRAG framework for interpretable, high-precision retrieval workfl… | 41 | 3960 | active |
| Lightning-AI/LitServe LitServe is a Python framework for building custom AI inference servers with full control over batching, routing, streaming, and scaling lo… | 92 | 3956 | active |
| thu-ml/SageAttention SageAttention is a family of quantized attention kernels (INT8/FP8/FP4) that accelerate transformer inference 2-5x over FlashAttention with… | 38 | 3956 | active |
| OpenHealthForAll/open-health OpenHealth is a self-hostable AI health assistant that consolidates personal health data (blood tests, checkups, wearables, symptoms) into … | 47 | 3954 | active |
| Nunchaku Nunchaku is a high-performance inference engine for 4-bit quantized diffusion models (and LLMs) based on the SVDQuant technique from an ICL… | 73 | 3953 | active |
| Anthropic Python SDK The official Python SDK for accessing the Anthropic Claude API. It provides a typed client for creating messages, streaming responses, and … | 93 | 3951 | stable |
| Paper2Poster/Paper2Poster Paper2Poster is an open-source multi-agent system (PosterAgent) that converts scientific paper PDFs into editable PowerPoint posters using … | 55 | 3951 | active |
| TransformerLensOrg/TransformerLens TransformerLens is a Python library for mechanistic interpretability of GPT-style transformer language models. It lets researchers load tho… | 98 | 3949 | active |
| duolahypercho/codex-router A local model router that lets external LLM providers (Anthropic, Kimi, DeepSeek, xAI, GitHub Copilot, and others) be used inside the Codex… | 79 | 3949 | active |
| lemony-ai/cascadeflow Cascadeflow is an open-source agent runtime intelligence layer (Python and TypeScript SDKs) that sits inside the AI agent loop to observe, … | 73 | 3941 | active |
| evalstate/fast-agent fast-agent is a Python CLI-first framework for building, running, and evaluating LLM agents with first-class MCP, ACP, and A2A protocol sup… | 84 | 3931 | active |
| huggingface/smollm Hugging Face's repository for the SmolLM and SmolVLM families of compact, fully open language and vision-language models, including trainin… | 67 | 3926 | active |
| snap-stanford/Biomni Biomni is a general-purpose biomedical AI agent from Stanford that autonomously executes research tasks across biomedical subfields. It com… | 68 | 3923 | active |
| ilysenko/codex-desktop-linux An unofficial community-maintained Linux distribution of OpenAI's ChatGPT desktop app (formerly Codex), repackaged from the official signed… | 61 | 3893 | active |
| LazyAGI/LazyLLM LazyLLM is a low-code Python framework for building multi-agent LLM applications, covering prototype assembly, data feedback, and iterative… | 92 | 3889 | active |
| synthetic-sciences/openscience OpenScience is an open-source AI workbench for scientific research that runs the full research loop: literature review, hypothesis formatio… | 75 | 3876 | active |
| CookSleep/gpt_image_playground A web-based playground for generating and editing images via the OpenAI gpt-image-2 API, built with React, TypeScript, and Tailwind CSS. It… | 77 | 3872 | active |
| NVlabs/VILA VILA is a family of open-source vision language models (VLMs) optimized for efficient video and multi-image understanding, spanning edge, d… | 54 | 3865 | active |
| thu-ml/TurboDiffusion TurboDiffusion is a Python framework that accelerates end-to-end video diffusion model generation by 100-200x using SageAttention, Sparse-L… | 58 | 3851 | active |
| HKUDS/Paper2Slides Paper2Slides is a Python application that converts research papers, reports, and documents into professional presentation slides and poster… | 50 | 3840 | active |
| predibase/lorax LoRAX is a multi-LoRA inference server that serves thousands of fine-tuned LLM adapters on a single GPU by sharing a common base model. It … | 60 | 3835 | active |
| jmiao24/Paper2Agent Paper2Agent is a multi-agent AI system that automatically converts research paper codebases into interactive AI agents with minimal human i… | 61 | 3831 | active |
| Open-Less/openless OpenLess is an open-source cross-platform voice input application that lets users hold a hotkey, speak, and have AI-polished text inserted … | 79 | 3811 | active |
| tbxark/ChatGPT-Telegram-Workers A Telegram ChatGPT bot deployable on Cloudflare Workers, Vercel, or Docker with a single-file, dependency-free setup. It supports multiple … | 95 | 3805 | active |
| SciSharp/LLamaSharp LLamaSharp is a cross-platform C#/.NET library that wraps llama.cpp to run LLaMA-family and other LLMs (including multimodal LLaVA) efficie… | 90 | 3803 | active |
| Polymarket/agents A Python developer framework and utility set for building AI agents that trade autonomously on Polymarket prediction markets. It integrates… | 10 | 3803 | active |
| 1186258278/OpenClawChineseTranslation A Chinese-localized distribution of OpenClaw, an open-source personal AI assistant that connects Claude/ChatGPT/Gemini LLMs to messaging pl… | 78 | 3796 | active |
| meta-llama/codellama Official Meta repository with inference code for the Code Llama family of large language models for code, based on Llama 2. It provides min… | 10 | 16251 | maintenance |
| AminBlg/SimpleEnglish An agent skill (SKILL.md) that makes LLMs write documentation in ASD-STE100 Simplified Technical English, the controlled language used in a… | 79 | 3794 | active |
| microsoft/mcp Microsoft's official catalog and monorepo of MCP (Model Context Protocol) server implementations, including Azure MCP and Microsoft Fabric … | 84 | 3788 | active |
| gofireflyio/aiac aiac is a command line tool and Go library that uses LLM providers such as OpenAI, Amazon Bedrock, and Ollama to generate Infrastructure-as… | 52 | 3786 | active |
| PeterH0323/Streamer-Sales Streamer-Sales is a fine-tuned LLM-based AI sales livestreamer application that generates persuasive product commentary from product descri… | 17 | 3775 | active |
| 1rgs/claude-code-proxy A Python proxy server that translates Anthropic API requests so Anthropic clients like Claude Code can run against OpenAI, Gemini, or Anthr… | 56 | 3753 | active |
| zou-group/textgrad TextGrad is a Python framework that implements automatic 'differentiation' via text, using LLM-generated natural language feedback as gradi… | 26 | 3750 | active |
| yaojingang/GEOFlow GEOFlow is an open-source content engineering and multi-site distribution system built for GEO (Generative Engine Optimization), combining … | 79 | 3747 | active |
| google-antigravity/antigravity-sdk-python A Python SDK for building AI agents powered by Google Antigravity and Gemini, abstracting the agentic loop behind a stateful, secure infras… | 59 | 3744 | active |
| shobrook/adrenaline Adrenaline is an AI-powered assistant that answers technical questions about programming concepts, GitHub repositories, documentation, and … | 32 | 3741 | active |
| PaddlePaddle/FastDeploy FastDeploy is a high-performance inference and deployment toolkit for large language models and vision-language models built on PaddlePaddl… | 82 | 3719 | active |
| llm-workflow-engine/llm-workflow-engine LLM Workflow Engine (LWE) is a power CLI and workflow manager for interacting with LLMs like ChatGPT/GPT-4 from the terminal. It supports m… | 86 | 3715 | active |
| memodb-io/Acontext Acontext is an open-source memory layer for AI agents that captures learnings from agent runs and stores them as human-readable Markdown sk… | 65 | 3691 | active |
| Stability-AI/StableLM StableLM is Stability AI's repository of open-weight decoder-only transformer language models, including the 3B-parameter StableLM-3B-4E1T … | 30 | 15671 | maintenance |
| AmintaCCCP/GithubStarsManager An AI-powered desktop and web application for managing GitHub stars, forks, and Gists with semantic vector search, automatic categorization… | 86 | 3667 | active |
| PublicAffairs/openai-gemini A serverless proxy that translates OpenAI API requests into Google Gemini API calls, letting any OpenAI-only tool use Gemini's free tier. I… | 53 | 3664 | active |
| Yue-Yang/ChatGPT-Siri A set of Apple Shortcuts that connect Siri to the ChatGPT API, turning Siri into an AI chat assistant on iPhone, iPad, Mac, HomePod, and Ca… | 69 | 3650 | active |
| zai-org/ChatGLM2-6B ChatGLM2-6B is an open-source bilingual (Chinese-English) 6B-parameter conversational large language model built on the GLM architecture. I… | 29 | 15505 | maintenance |