Open-source llm-inference software

3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.

3358 products, primary matches first, then adoption-weighted; health v2 shown.

Product	Health v2	Stars	Maturity
HKUDS/AutoAgent AutoAgent is a fully-automated, zero-code framework for creating and deploying LLM agents using natural language alone. It acts as an auton…	39	9806	active
tmc/langchaingo Langchaingo is the Go language port of LangChain, providing composable building blocks for writing LLM-powered applications in Go. It offer…	87	9712	active
mshumer/gpt-prompt-engineer A Jupyter Notebook-based tool that automatically generates, tests, and ranks LLM prompts for a given task using GPT-4, GPT-3.5-Turbo, or Cl…	47	9681	active
xorbitsai/inference Xinference is an open-source model serving platform for deploying LLMs, embedding, speech, image, and multimodal models via a unified OpenA…	93	9616	active
Mobilerun Mobilerun is an open-source Python framework for controlling Android and iOS devices with LLM-powered agents using natural language command…	82	9613	active
zhouxiaoka/autoclip AutoClip is a self-hosted AI-powered video clipping system that downloads videos from YouTube and Bilibili, uses LLM-based content analysis…	86	9577	active
Spring AI Spring AI is an application framework for AI engineering that provides Spring-friendly, portable abstractions for integrating AI models int…	94	9542	active
fastrepl/anarlog Anarlog is an open-source, local-first AI meeting notepad and Granola alternative that records device audio without a bot joining the call,…	85	9527	active
BAML BAML is a domain-specific programming language for building LLM agents, with a Rust-like type system, typed errors, built-in tests/evals, a…	92	9423	active
oumi-ai/oumi Oumi is an open-source Python framework and platform for the end-to-end lifecycle of open-weight LLMs: data synthesis, fine-tuning (SFT, Lo…	91	9407	active
Future-House/paper-qa PaperQA2 is a Python library and CLI for high-accuracy retrieval augmented generation (RAG) over PDFs, text, Office documents, and source c…	95	9365	active
LearningCircuit/local-deep-research A self-hosted AI research assistant that performs deep, iterative multi-source research using local or cloud LLMs (Ollama, llama.cpp, OpenA…	82	9359	active
alvinunreal/oh-my-opencode-slim A lightweight agent orchestration plugin for OpenCode that delegates tasks to a team of seven specialized sub-agents (Orchestrator, Explore…	78	9344	active
shaxiu/XianyuAutoAgent An AI-powered customer service bot for the Xianyu (Goofish) secondhand marketplace that provides 24/7 automated message handling. It uses L…	56	9275	active
MervinPraison/PraisonAI PraisonAI is a Python multi-agent framework for building and deploying autonomous, self-improving AI agents with built-in memory, RAG, and …	89	9274	active
modelscope/modelscope ModelScope is a Python library and ecosystem built on the 'Model-as-a-Service' concept, providing unified APIs to download, run inference o…	96	9149	active
EvoMap/evolver Evolver is a Node.js-based self-evolving engine for AI agents built on the GEP (Gene Evolution Protocol), letting agents turn experience in…	77	9145	active
Nutlope/aicommits Aicommits is a Node.js CLI that automatically generates git commit messages from your staged changes using AI. It supports multiple LLM pro…	94	9100	active
FareedKhan-dev/kimi-k3-in-c A dependency-free C99 inference engine that runs the 2.78-trillion-parameter Kimi K3 model on a single CPU with as little as 8 GB of RAM by…	79	9033	active
deepseek-ai/DeepGEMM DeepGEMM is a high-performance CUDA BLAS kernel library for NVIDIA tensor cores, providing FP8, FP4, and BF16 GEMMs plus fused MoE and othe…	82	8937	active
Anxcye/anx-reader Anx Reader is a cross-platform e-book reader app built with Flutter that supports EPUB, MOBI, AZW3, FB2, TXT, and PDF formats. It integrate…	81	8922	active
Rig Rig is a Rust library for building modular, scalable LLM-powered applications and agents, offering a unified API across 20+ model providers…	87	8911	active
iflytek/astron-agent Astron Agent is an enterprise-grade, commercial-friendly agentic workflow platform for building, orchestrating, and deploying multi-agent A…	83	8910	active
BentoML BentoML is a Python framework for building online serving systems for AI apps and model inference, turning model inference scripts into RES…	91	8909	stable
Strands Agents Strands Agents is an open-source SDK (Python and TypeScript) for building production AI agents with a model-driven approach, supporting any…	83	8884	active
elder-plinius/OBLITERATUS OBLITERATUS is a Python toolkit for abliteration — identifying and removing refusal behaviors from large language models by locating and in…	79	8861	active
Nutlope/logocreator An open-source AI logo generator web app that creates brand-ready logos using FLUX models on Together AI, with logo editing, style presets,…	62	8771	active
freecodexyz/free-code A community fork of Anthropic's Claude Code terminal AI coding agent, rebuilt from an exposed source snapshot with telemetry removed, secur…	54	8757	active
a-ghorbani/pocketpal-ai PocketPal AI is an open-source mobile app that runs GGUF language models entirely on-device for private, offline chat. It supports model do…	87	8727	active
google-deepmind/alphafold3 Google DeepMind's official implementation of the AlphaFold 3 inference pipeline for predicting biomolecular structures and interactions. It…	81	8643	active
HKUDS/ClawWork ClawWork is an open-source benchmark and framework that turns AI assistants into 'AI coworkers' which must earn income by completing real p…	44	8560	active
OpenBMB/XAgent XAgent is an open-source, LLM-driven autonomous agent framework for automatically solving complex tasks. It features a dispatcher and plann…	58	8551	active
reorproject/reor Reor is a local-first, AI-powered desktop note-taking app that automatically links related notes, answers questions over your notes via RAG…	10	8549	active
bitsandbytes-foundation/bitsandbytes bitsandbytes is a Python library providing k-bit quantization primitives for PyTorch, enabling 8-bit (LLM.int8()) and 4-bit (QLoRA) quantiz…	94	8514	active
OptimalScale/LMFlow LMFlow is an extensible Python toolkit for finetuning and inference of large foundation models such as LLaMA, GPT-2, and Galactica. It prov…	61	8487	active
OGX OGX (formerly Llama Stack) is an open-source, OpenAI-compatible agentic API server and Python library that bundles inference, vector stores…	87	8441	active
qualcomm/GenieX GenieX is an on-device generative AI inference runtime for Qualcomm Snapdragon devices, running frontier LLMs and VLMs on the Hexagon NPU, …	87	8440	active
AAswordman/Operit Operit AI is an open-source AI agent and chat application for Android built with Kotlin and Jetpack Compose. It connects cloud or local LLM…	86	8416	active
withastro/flue Flue is an open-source TypeScript framework for building autonomous AI agents using a React-like hooks API ('use agent' directive, useModel…	77	8415	active
Dynamo NVIDIA Dynamo is an open-source, datacenter-scale distributed inference serving framework that orchestrates inference engines like vLLM, SG…	88	8345	active
zai-org/ChatGLM-6B ChatGLM-6B is an open-source bilingual (Chinese-English) 6.2B-parameter dialogue language model based on the GLM architecture, optimized fo…	31	40938	maintenance
anthropics/claude-agent-sdk-python The official Python SDK for building agents on top of Claude Code, providing async query APIs, bidirectional conversations, custom tools vi…	83	8311	active
duixcom/Duix-Mobile Duix Mobile is an open-source SDK for building real-time interactive AI avatars (digital humans) that run on-device on Android, iOS, tablet…	63	8267	active
k8sgpt k8sgpt is a CLI tool that scans Kubernetes clusters, diagnoses issues, and triages them in plain English using AI backends like OpenAI, Azu…	98	8255	active
mini-swe-agent A radically minimal AI software engineering agent (~100 lines of core Python) that solves GitHub issues and assists in the command line, sc…	77	8226	active
mini-swe-agent SWE-agent is a Python library that lets a large language model autonomously fix GitHub issues, discover security vulnerabilities, or run cu…	77	8226	active
intitni/CopilotForXcode An Xcode Source Editor Extension that brings GitHub Copilot, Codeium, and ChatGPT-powered AI assistance to Xcode on macOS. It provides inli…	73	8208	active
OpenDCAI/DataFlow DataFlow is a Python framework for preparing, cleaning, and synthesizing data for LLM training and RAG using composable LLM-based operators…	77	8201	active
AI4Finance-Foundation/FinRobot FinRobot is an open-source AI agent platform for financial applications built on large language models, unifying LLMs, reinforcement learni…	78	8140	active
NVIDIA/Isaac-GR00T NVIDIA Isaac GR00T N1.7 is an open vision-language-action (VLA) foundation model for generalized humanoid robot skills, taking language and…	70	8116	stable
InternLM/lmdeploy LMDeploy is a toolkit for compressing, quantizing, deploying, and serving large language models, built around its high-performance TurboMin…	96	8107	active
lm-sys/FastChat FastChat is an open platform for training, serving, and evaluating large language model based chatbots, originally releasing the Vicuna mod…	54	39548	maintenance
mattpocock/sandcastle A TypeScript library for orchestrating AI coding agents (like Claude Code) inside isolated sandboxes via a single sandcastle.run() call. It…	69	8073	active
Osaurus Osaurus is a native Swift macOS application that acts as a local-first AI harness, running open models on Apple Silicon (via MLX and Apple …	82	8060	active
PriorLabs/TabPFN TabPFN is a pre-trained tabular foundation model (transformer) that performs classification, regression, forecasting, and anomaly detection…	96	8048	active
RayVentura/ShortGPT ShortGPT is a Python framework that automates AI-driven video content creation, combining LLM script generation, voiceover synthesis, asset…	31	7973	active
OpenCoworkAI/open-codesign Open CoDesign is an open-source, local-first desktop AI design tool that turns prompts into polished artifacts like UI prototypes, slide de…	79	7967	active
Upsonic/Upsonic Upsonic is a Python framework for building autonomous and traditional AI agents, with support for tools, memory, knowledge bases, RAG, mult…	71	7956	active
ArvinLovegood/go-stock A Wails + NaiveUI desktop application that combines stock market data (A-shares, Hong Kong, US markets) with LLM-powered analysis, includin…	85	7891	active
chenyme/grok2api A self-hosted multi-account API gateway for Grok Build, Grok Web, and Grok Console, written in Go with a React frontend. It exposes Grok ca…	82	7797	active
2FastLabs/agent-squad Agent Squad is an open-source framework for orchestrating multiple AI agents, routing each user query to the most suitable specialized agen…	89	7780	active
EricLBuehler/mistral.rs mistral.rs is a fast and flexible LLM inference engine written in Rust, supporting many model families with quantization (ISQ/UQFF, GGUF), …	92	7741	active
leejet/stable-diffusion.cpp A pure C/C++ inference engine for diffusion models (Stable Diffusion, FLUX, Wan, Qwen Image, Z-Image, and more) built on ggml in the style …	92	7734	active
sweepai/sweep Sweep is an AI coding assistant plugin for JetBrains IDEs (IntelliJ, PyCharm, GoLand, etc.) offering an AI agent, fast next-edit autocomple…	36	7708	active
farzaa/clicky Clicky is an open-source macOS app that acts as an AI teacher living next to your cursor — it can see your screen, talk with you via voice,…	47	7707	active
meta-llama/llama-models Meta's official repository of Llama large language models with Python utilities for working with them, including model cards, licenses, and…	53	7693	active
mufeedvh/code2prompt Code2Prompt is a Rust-based CLI tool that converts an entire codebase into a single structured LLM prompt, including a source tree, file co…	66	7688	active
tailcallhq/forgecode ForgeCode is an open-source, Rust-based CLI coding agent (AI pair programmer) that integrates LLMs from 300+ providers directly into the te…	85	7637	active
openJiuwen-ai/jiuwenswarm JiuwenSwarm is an open-source multi-agent AI system built on openJiuwen that lets a Leader agent decompose complex tasks and coordinate tea…	82	7602	active
pickle-com/glass Glass by Pickle is an open-source Electron desktop app that acts as a 'digital mind extension', seeing your screen and listening in real ti…	39	7602	active
lllyasviel/Omost Omost is a Python application that converts LLM coding capability into image composition by having pretrained LLMs (based on Llama3 and Phi…	25	7595	active
GoogleCloudPlatform/kubectl-ai kubectl-ai is an AI-powered Kubernetes assistant that translates natural-language user intent into precise kubectl operations. It is a Go C…	71	7587	active
lingfengQAQ/webnovel-writer A Claude Code plugin for long-form serialized webnovel writing, built in Python. It provides slash-command skills for initializing story se…	84	7566	active
mnfst/manifest Manifest is an open-source LLM gateway and router that lets AI agents and apps reach all their model providers through one OpenAI- and Anth…	93	7551	active
di-sukharev/opencommit OpenCommit is a CLI tool that uses LLMs (OpenAI, Claude, Ollama, and other providers) to automatically generate meaningful git commit messa…	66	7545	active
Refly Refly is an open-source platform for building, versioning, and running reusable AI agent skills through a natural-language 'vibe workflow' …	74	7531	active
guardrails-ai/guardrails Guardrails is a Python framework for building reliable LLM applications by running Input/Output Guards that detect and mitigate risks, and …	95	7527	active
teamchong/pxpipe pxpipe is a local TypeScript proxy that reduces Claude Code input token usage by rewriting bulky text context (system prompts, tool docs, h…	81	7525	active
algorithmicsuperintelligence/openevolve OpenEvolve is an open-source Python implementation of DeepMind's AlphaEvolve, an evolutionary coding agent that uses LLM ensembles to auton…	86	7518	active
microsoft/graphrag GraphRAG is a Python library and CLI pipeline from Microsoft Research that extracts a knowledge graph from unstructured text, builds commun…	92	36292	maintenance
zai-org/GLM-OCR GLM-OCR is an open-source 0.9B-parameter multimodal OCR model built on the GLM-V encoder-decoder architecture for complex document understa…	73	7504	active
open-compass/opencompass OpenCompass is a Python-based evaluation platform and toolkit for benchmarking large language models across 100+ datasets, supporting model…	93	7471	active
apple/ml-fastvlm Official implementation of FastVLM, a vision language model with an efficient hybrid vision encoder (FastViTHD) that reduces token count an…	62	7423	active
MuiseDestiny/zotero-gpt A Zotero plugin that integrates OpenAI GPT models into the Zotero reference manager, letting users ask questions about PDFs, summarize pape…	73	7420	active
spinabot/brigade Brigade is a self-hosted, MIT-licensed multi-agent runtime where a crew of AI agents on an org chart delegate work, switch models mid-task,…	73	7412	active
vladmandic/sdnext SD.Next is an open-source, self-hosted WebUI server application for AI generative image and video creation built on Stable Diffusion and Di…	77	7353	active
zai-org/GLM-5 GLM-5 is a family of open-weight large language models from Z.ai (Zhipu AI), including the flagship GLM-5.2 with 1M-token context and the m…	60	7329	active
interviewstreet/hiring-agent An open-source AI agent pipeline from HackerRank that extracts structured data from resume PDFs, enriches it with GitHub signals, and produ…	63	7312	active
kyegomez/swarms Swarms is an enterprise-grade Python framework for orchestrating multi-agent systems powered by large language models, offering prebuilt se…	66	7282	active
Paddle Lite Paddle Lite is a high-performance, lightweight deep learning inference engine from Baidu's PaddlePaddle ecosystem, designed for mobile, emb…	55	7277	active
JerryZLiu/Dayflow Dayflow is an open-source, local-first macOS app that automatically turns your screen activity into a timeline of what you accomplished, us…	81	7276	active
InternLM/InternLM Official repository for the InternLM series of open-source large language models (InternLM, InternLM2, InternLM2.5, InternLM3), including p…	38	7276	active
mit-han-lab/streaming-llm StreamingLLM is a research framework from MIT Han Lab implementing the Attention Sinks method (ICLR 2024) for efficient streaming language …	28	7267	stable
ml-explore/mlx-lm A Python package for running and fine-tuning large language models on Apple silicon using the MLX framework. It integrates with the Hugging…	78	7259	active
magnitudedev/magnitude Magnitude is an open-source CLI tool and agent that bundles local LLM inference with an agent harness, profiling your hardware to recommend…	82	7240	active
MineDojo/Voyager Voyager is an open-source LLM-powered embodied agent that plays Minecraft through GPT-4, using an automatic curriculum, an ever-growing ski…	30	7219	active
deepseek-ai/DeepSpec DeepSpec is a full-stack Python codebase from DeepSeek for training and evaluating draft models used in speculative decoding of large langu…	51	7207	active
NVIDIA-NeMo/Guardrails NVIDIA NeMo Guardrails is an open-source Python toolkit for adding programmable guardrails to LLM-based conversational applications. It sit…	90	7181	active
Automaton Automaton is a TypeScript runtime for continuously running, self-improving AI agents that pay for their own compute via an Ethereum wallet …	67	7156	active
dwzhu-pku/PaperBanana PaperBanana is a reference-driven multi-agent framework that automatically generates publication-ready academic illustrations such as metho…	52	7150	active