Open-source llm-inference software

3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.

3358 products, primary matches first, then adoption-weighted; health v2 shown.

Product	Health v2	Stars	Maturity
openai/openai-python The official OpenAI Python library provides a convenient wrapper around the OpenAI REST API. It supports both synchronous and asynchronous …	96	31870	active
stanford-oval/storm STORM is an LLM-powered system that researches a topic via internet search and generates full-length, cited, Wikipedia-like reports. It is …	32	31520	active
ScrapeGraphAI ScrapeGraphAI is a Python library that uses LLMs and direct graph logic to build web scraping pipelines for websites and local documents (X…	87	31358	active
decolua/9router 9Router is a self-hosted AI gateway/router that connects AI coding tools like Claude Code, Cursor, Codex, and Cline to 60+ LLM providers th…	74	30634	active
DeepAgents Deep Agents is an open-source, batteries-included agent harness from LangChain built on LangGraph, providing an opinionated agent with file…	82	30425	active
tobi/qmd QMD is a local, on-device CLI search engine that indexes markdown notes, docs, meeting transcripts, and knowledge bases. It combines BM25 f…	80	30141	active
sipeed/picoclaw PicoClaw is an ultra‑lightweight AI assistant written in Go that runs on $10 hardware with less than 10 MB RAM. It provides a self‑containe…	76	30013	active
GPT Researcher GPT Researcher is a Python‑based autonomous AI agent that performs deep, multi‑source research and generates detailed reports with citation…	95	29943	active
modular/modular Modular Platform is an open‑source stack that includes the Mojo systems programming language and the MAX framework for building, optimizing…	90	29847	active
openai/openai-agents-python OpenAI Agents SDK is a lightweight Python framework for building multi-agent LLM workflows with agents, handoffs, tools, guardrails, sessio…	82	29800	active
OpenDataLoader PDF OpenDataLoader PDF is an open‑source PDF parsing library that converts PDFs into AI‑ready formats such as Markdown, JSON with bounding boxe…	83	29741	active
alibaba/page-agent Page Agent is a JavaScript library that embeds an AI GUI agent directly into any web page with a single script, letting users control the i…	82	29343	active
Mastra Mastra is a modern TypeScript framework for building AI-powered applications and autonomous agents. It provides model routing, workflow orc…	87	28892	active
MLX MLX is an array computation framework for machine learning on Apple silicon, developed by Apple ML research. It offers NumPy-like Python AP…	95	28839	active
Crush Crush is a terminal‑based coding assistant that connects to various LLMs via OpenAI‑compatible APIs. It provides session‑aware, LSP‑enhance…	83	28696	active
Crush Crush is a terminal-based agentic coding assistant from Charm that wires your codebase, tools, and workflows into the LLM of your choice. I…	83	28696	active
firecrawl/open-lovable Open Lovable is an AI-powered web application that lets users chat with an LLM to clone and recreate any website as a modern React app. It …	63	28681	active
Semantic Kernel Semantic Kernel is a model‑agnostic SDK that lets developers integrate large language models and build AI agents quickly. It provides tools…	90	28582	active
QwenLM/qwen-code Qwen Code is an open‑source AI coding agent that runs in your terminal and can be extended to IDEs, desktop apps, and browsers. It leverage…	82	28566	active
srbhr/Resume-Matcher Resume Matcher is an AI‑powered application that helps you create, tailor, and export resumes and cover letters for specific job postings. …	87	28535	active
Kilo Kilo is an open‑source AI coding agent that works across VS Code, JetBrains IDEs and the command line. It lets developers pick from 500+ mo…	84	27653	active
Kilo Kilo Code is an open-source (MIT) AI coding agent available as a VS Code extension, JetBrains plugin, and CLI, with cloud agents and a mode…	84	27653	active
virattt/dexter Dexter is an autonomous AI agent built in TypeScript that performs deep financial research by planning tasks, executing them with real-time…	82	27647	active
TencentCloud/TencentDB-Agent-Memory TencentDB Agent Memory is a team-level memory hub that stores and serves conversational, document, and code embeddings for AI agents. It pr…	79	27532	active
Fosowl/agenticSeek AgenticSeek is a fully local, privacy-focused AI assistant that serves as an open-source alternative to Manus AI. It runs reasoning models …	64	27482	active
vercel/ai Vercel AI is a TypeScript SDK that provides a unified API for interacting with large language models across many providers. It includes uti…	93	27251	active
baidu/Unlimited-OCR Baidu's Unlimited-OCR is an open vision-language OCR model for one-shot long-horizon document parsing, extending DeepSeek-OCR. It provides …	53	26763	active
Haystack Haystack is an open‑source Python framework for orchestrating LLM‑powered pipelines, agents and RAG applications. It provides modular compo…	99	26720	active
MiniCPM MiniCPM-V and MiniCPM-o are a series of small multimodal large language models for efficient image, video, and audio understanding, deploya…	59	26452	active
yikart/AiToEarn AiToEarn is an Electron‑based desktop application that uses AI agents to automate content creation, publishing, and monetization across mul…	80	26265	active
mozilla-ai/llamafile Llamafile packages LLMs and Whisper models into a single, portable executable. It runs locally on most operating systems without any instal…	90	26224	active
black-forest-labs/flux Official minimal inference code from Black Forest Labs for running FLUX.1 open-weight image generation and editing models. It supports text…	35	26009	active
SYSTRAN/faster-whisper A fast reimplementation of OpenAI's Whisper speech-to-text model built on the CTranslate2 inference engine, offering up to 4x speedup and l…	52	25631	active
MiniGPT-4 MiniGPT-4 is an open-source Python library that aligns a frozen visual encoder with a frozen large language model to enable multimodal imag…	30	25605	active
TOON TOON (Token-Oriented Object Notation) is a compact, human-readable encoding of the JSON data model designed to reduce token usage in LLM pr…	78	25440	active
microsoft/JARVIS JARVIS (HuggingGPT) is a system that uses an LLM as a controller to plan tasks and orchestrate numerous expert ML models from the HuggingFa…	42	25307	active
titanwings/distilly Distilly is a Python tool that distills a person's experience, judgment, voice, and working style from messages, documents, interviews, and…	59	25182	active
vxcontrol/pentagi PentAGI is a Go-based autonomous AI agent system for automated penetration testing. It provides a web UI, GraphQL API, and integrates with …	76	25181	active
haotian-liu/LLaVA LLaVA (Large Language and Vision Assistant) is an open-source multimodal large language model framework implementing visual instruction tun…	21	25067	active
google-ai-edge/gallery Google AI Edge Gallery is a mobile and desktop app that lets you run large language models and multimodal AI directly on your device. It pr…	87	25018	active
deepseek-ai/DeepSeek-Coder DeepSeek Coder is a series of open code language models (1B to 33B) trained from scratch on 2T tokens of code and natural language, with a …	47	24295	active
deepseek-ai/DeepSeek-OCR DeepSeek-OCR is an open vision-language model from DeepSeek AI that researches 'contexts optical compression' - encoding long text contexts…	43	23965	active
Tencent/ncnn ncnn is a high-performance neural network inference framework written in C++ and optimized for mobile, embedded, and desktop deployment. It…	83	23907	stable
vanna-ai/vanna Vanna is an open-source Python framework for building user-aware AI agents that translate natural-language questions into SQL queries, char…	10	23812	active
sinaptik-ai/pandas-ai PandasAI is a Python library that lets users query data in SQL, CSV, or parquet format using natural language, powered by LLMs and RAG. It …	52	23808	active
antirez/ds4 DwarfStar (ds4) is a native inference engine for running large language models like DeepSeek V4 Flash and GLM 5.x on consumer hardware. It …	57	23620	active
HKUDS/RAG-Anything RAG-Anything is a Python-based, all‑in‑one Retrieval‑Augmented Generation framework that supports text, images, tables, and math. It provid…	86	23584	active
Skyvern-AI/skyvern Skyvern is an open‑source platform that adds AI and computer‑vision capabilities to Playwright for automating browser‑based workflows. It p…	89	23226	active
mlc-ai/mlc-llm MLC LLM is a machine learning compiler and high-performance deployment engine for large language models, built on TVM-based ML compilation.…	61	23176	active
HKUDS/AI-Trader AI-Trader is an agent-native trading platform where autonomous AI agents register, trade, and compete with live mark-to-market scoring and …	53	22799	active
NemoClaw NVIDIA NemoClaw is an open-source reference stack for running AI coding agents such as OpenClaw, Hermes, and LangChain Deep Agents Code sec…	69	22754	active
wandb/openui OpenUI is an open-source web application from Weights & Biases that generates UI components from natural language descriptions, rendering t…	69	22573	active
PromtEngineer/localGPT LocalGPT is a fully private, on-premise document intelligence platform that lets users chat with, summarize, and query their own documents …	71	22197	active
ONNX Runtime ONNX Runtime is a cross-platform, high-performance machine-learning accelerator for running inference and training on ONNX models. It suppo…	100	22115	stable
PrimeIntellect-ai/prime-agent Prime Agent is an open‑source autonomous coding and research assistant built in Rust. It runs as a persistent REPL that can call sub‑agents…	82	22065	active
Guidance Guidance is a Python library that provides a domain‑specific language for steering large language models. It lets developers embed conditio…	73	21799	active
Guidance Guidance is a Python library providing an efficient programming paradigm for steering large language models, letting you interleave control…	73	21799	active
huggingface/peft 🤗 PEFT is a Python library that provides state‑of‑the‑art parameter‑efficient fine‑tuning methods for large pretrained models. It integrate…	95	21795	active
KKKKhazix/khazix-skills Khazix Skills is a Python‑based collection of ready‑to‑use AI Agent Skills that follow the open Agent Skills standard. Each skill is a stru…	59	21494	active
huggingface/candle Candle is a minimalist machine learning framework for Rust focused on performance and ease of use, with CPU and CUDA GPU support. It ships …	74	21182	active
pydantic/pydantic-ai Pydantic AI is a typed Python SDK for building AI agents and LLM applications. It provides an extensible agent loop, multi‑modal capabiliti…	87	20670	active
openai/gpt-oss gpt-oss provides open-weight large language models (120B and 20B) with full chain‑of‑thought reasoning and native agentic capabilities. The…	66	20428	active
karpathy/llama2.c A minimal, dependency-free implementation of the Llama 2 LLM architecture that trains in PyTorch and runs inference in a single ~700-line p…	29	20192	active
Alibaba-NLP/DeepResearch Tongyi DeepResearch is an open-source agentic large language model (30.5B total parameters, 3.3B activated per token) plus training and inf…	49	20014	active
QwenLM/Qwen3-VL Qwen3-VL is a series of open-weight multimodal vision-language models from Alibaba's Qwen team, available in Dense and MoE architectures wi…	49	19998	active
kvcache-ai/ktransformers KTransformers is a Python library that provides a high‑performance framework for heterogeneous CPU‑GPU inference and fine‑tuning of large l…	91	19618	active
openai/evals OpenAI Evals is a Python framework for evaluating large language models and LLM-based systems, with an open-source registry of benchmark ev…	63	19500	active
nari-labs/dia Dia is a 1.6B-parameter open-weight text-to-speech model from Nari Labs that generates ultra-realistic multi-speaker dialogue in a single p…	40	19401	active
WebLLM WebLLM is a high‑performance in‑browser LLM inference engine written in TypeScript. It runs large language models directly in the browser u…	90	19160	active
lss233/kirara-ai Kirara AI is a self-hosted, DIY multimodal AI chatbot framework that connects large language models (DeepSeek, ChatGPT, Claude, Gemini, Oll…	43	19123	active
Huanshere/VideoLingo VideoLingo is an open‑source Python application that combines speech recognition, AI‑powered subtitle segmentation, translation and voice‑c…	89	18808	active
confident-ai/deepeval DeepEval is an open-source Python framework for evaluating LLM applications with pytest-style unit tests and 50+ research-backed metrics in…	95	18733	active
meta-llama/llama-cookbook The Llama Cookbook is a collection of Jupyter notebooks and scripts that demonstrate how to use Meta's Llama models for inference, fine‑tun…	54	18559	active
vercel-labs/json-render json-render is a Generative UI framework that lets AI generate dynamic, personalized interfaces from natural language prompts, constrained …	76	18472	active
Parlant Parlant is an open-source Python framework that acts as an interaction control harness for building reliable, customer-facing LLM agents. I…	78	18298	active
LangChain LangChain.js is a TypeScript framework for building LLM-powered applications, providing standard interfaces for agents, models, embeddings,…	95	18284	active
xming521/WeClone WeClone is an end-to-end Python framework for creating a personal AI digital twin by fine-tuning large language models on your exported cha…	78	18278	active
bradautomates/claude-video A Claude Code plugin and Agent Skill that lets AI coding agents watch videos. It downloads videos via yt-dlp, extracts scene-aware frames w…	79	18248	active
avante-corp/avante.nvim A Neovim plugin that emulates the Cursor AI IDE experience inside the editor. It provides AI-driven code suggestions, chat-based code assis…	89	18175	active
DeepWiki / Grok Wiki DeepWiki-Open (now Grok-Wiki) is an open-source, AI-powered tool that automatically generates interactive wikis, documentation, diagrams, a…	62	18138	active
apple/ml-stable-diffusion Apple's official tooling for running Stable Diffusion on Apple Silicon using Core ML. It includes a Python package for converting PyTorch m…	65	17965	active
Toonflow Toonflow is an open-source cross-platform desktop application that turns novels and scripts into animated short dramas using AI. It integra…	83	17894	active
ahmedkhaleel2004/gitdiagram GitDiagram is a web application that turns any public or private GitHub repository into an interactive architecture diagram in seconds, wit…	66	17880	active
camel-ai/camel CAMEL is an open-source Python framework for building multi-agent AI systems, providing agents, role-based societies, memory, RAG pipelines…	82	17834	active
WrenAI WrenAI is an open-source generative BI (GenBI) engine that turns natural-language questions into governed SQL, charts, and dashboards acros…	87	17826	active
deepseek-ai/Janus Janus-Series is DeepSeek's family of unified multimodal models (Janus, Janus-Pro, JanusFlow) that combine multimodal understanding and imag…	23	17755	active
lidge-jun/opencodex opencodex is a lightweight local proxy (installed via npm, running on a bundled Bun runtime) that translates OpenAI Codex's Responses API a…	77	17631	active
microsoft/data-formulator Data Formulator is an AI-powered interactive data analysis and visualization application from Microsoft that lets users connect data source…	82	17266	active
HKUDS/DeepCode DeepCode is an open agentic coding framework from HKUDS that orchestrates multi-agent LLM pipelines to autonomously generate production-gra…	87	16710	active
xbtlin/ai-berkshire A collection of AI research skills for Claude Code and Codex that systematizes the value investing methodologies of Buffett, Munger, Duan Y…	60	16597	active
stackblitz/bolt.new Bolt.new is an open-source AI-powered web development agent that lets users prompt, run, edit, and deploy full-stack web applications direc…	23	16568	active
ai-shifu/ChatALL ChatALL is an Electron desktop application that sends prompts to multiple AI chatbots (ChatGPT, Claude, Bing Chat, ChatGLM, ERNIE, and more…	72	16503	active
Novel Novel is an open-source, headless Notion-style WYSIWYG rich text editor built on Tiptap and ProseMirror, with AI-powered autocompletion via…	33	16451	active
Transformers.js Transformers.js is a JavaScript library that lets you run Hugging Face Transformers pretrained models directly in the browser (or Node.js) …	98	16349	active
andrewyng/aisuite aisuite is a lightweight Python library providing a unified, OpenAI-style Chat Completions API across many LLM providers (OpenAI, Anthropic…	81	16320	active
MNN MNN is a lightweight, high-performance deep learning inference engine developed by Alibaba, supporting LLMs, vision, audio, and multimodal …	90	16188	active
raga-ai-hub/RagaAI-Catalyst RagaAI Catalyst is a Python SDK for observability, monitoring, and evaluation of LLM and agentic AI applications, with tracing of agents, L…	52	16164	active
WEIFENG2333/VideoCaptioner VideoCaptioner is an open-source AI-powered video subtitling tool that handles the full pipeline: speech recognition, LLM-based subtitle se…	81	16144	active
GLips/Figma-Context-MCP Framelink Figma Context MCP is a Model Context Protocol server that fetches and simplifies Figma design data (layout, styling, components) …	82	15989	active
GreyDGL/PentestGPT PentestGPT is an AI-powered penetration testing agent framework that drives LLMs (Claude Code, Codex, or many providers in legacy mode) to …	65	15948	active