Open-source llm-inference software
3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
3358 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| airockchip/rknn-llm RKLLM is Rockchip's software stack for converting, quantizing, and running large language models on Rockchip NPU chips such as RK3588 and R… | 83 | 1704 | active |
| leochlon/hallbayes Berry is a local MCP server that detects hallucinations by measuring whether an LLM's claims are actually supported by the cited evidence, … | 54 | 1699 | active |
| qiutongxue/oba-live-tool A desktop Electron application that automates live-streaming e-commerce tasks on Chinese platforms including Douyin, Xiaohongshu, Kuaishou,… | 85 | 1695 | active |
| googleapis/js-genai The official Google Gen AI SDK for TypeScript and JavaScript, providing access to the Gemini Developer API and Vertex AI (Gemini Enterprise… | 85 | 1691 | active |
| hrishioa/lumentis Lumentis is an AI-powered CLI tool that generates comprehensive, easy-to-skim documentation websites from transcripts, notes, and unstructu… | 26 | 1690 | active |
| Subtitle Translator A cross-platform Electron desktop app that translates subtitle files (.ass, .srt, .ssa, .vtt) using LLMs like ChatGPT. It preserves context… | 88 | 1689 | active |
| devenjarvis/lathe Lathe is a Go CLI plus local web UI that generates hands-on, multi-part technical tutorials on demand using LLM skills run inside coding ag… | 72 | 1689 | active |
| Anemll/Anemll ANEMLL is an open-source Python library and toolchain for converting and running large language models on the Apple Neural Engine (ANE) via… | 70 | 1686 | active |
| nextify-limited/libra Libra AI is an open-source alternative to V0 and Lovable that generates production-grade full-stack web applications from natural language … | 34 | 1684 | active |
| intentee/paddler Paddler is an open-source LLM/VLM load balancer and serving platform built around the llama.cpp (ggml) ecosystem, packaged as a single Rust… | 87 | 1679 | active |
| turboderp-org/exllamav3 ExLlamaV3 is a Python library for fast quantization and inference of large language models on consumer-class GPUs, featuring the EXL3 quant… | 83 | 1678 | active |
| microsoft/Mage Mage is a family of lightweight 4B-parameter multimodal models from Microsoft, including Mage-VL for image and video understanding and Mage… | 55 | 1677 | active |
| elixir-nx/bumblebee Bumblebee is an Elixir library providing pre-trained neural network models built on Axon, with integration for downloading models from Hugg… | 94 | 1674 | active |
| deepsense-ai/ragbits Ragbits is a Python framework of modular building blocks for rapidly developing generative AI applications, covering LLM interaction, promp… | 68 | 1674 | active |
| Prat011/free-cluely An open-source Electron desktop app that acts as an invisible AI assistant, providing real-time answers and insights during meetings, inter… | 50 | 1673 | active |
| intel/auto-round AutoRound is Intel's advanced quantization toolkit for LLMs and vision-language models, using sign-gradient descent to achieve high accurac… | 94 | 1672 | active |
| LeapLabTHU/cooragent Cooragent is an AI agent collaboration community and framework where users can create powerful agents from a single sentence and have them … | 54 | 1671 | active |
| Tencent/AngelSlim AngelSlim is a Python toolkit from Tencent for compressing large language models and related architectures (VLMs, diffusion, audio models) … | 70 | 1669 | active |
| bytedance/pasa PaSa is an LLM-powered paper search agent from ByteDance that autonomously invokes search tools, reads papers, and selects relevant referen… | 28 | 1665 | active |
| xiaoY233/Chat2API Chat2API is an Electron-based desktop application that exposes OpenAI-compatible API endpoints backed by official AI provider web UIs such … | 65 | 1664 | active |
| TurboSeek TurboSeek is an open-source AI search engine web application inspired by Perplexity, built with Next.js and TypeScript. It searches the web… | 62 | 1661 | active |
| OpenScholar OpenScholar is a retrieval-augmented language model system from the Allen Institute for AI that answers scientific questions by searching t… | 35 | 1659 | active |
| LlamaEdge/LlamaEdge LlamaEdge is a lightweight Rust and WasmEdge-based runtime for running open-source LLMs locally or on edge devices, with CLI chat apps and … | 64 | 1658 | active |
| Agently Agently is a Python framework for building production-grade GenAI applications with structured outputs, observable actions, MCP/tool integr… | 92 | 1656 | active |
| LLM-Red-Team/kimi-cc A shell-based installer that configures Claude Code to run on Moonshot's Kimi K2 model instead of Anthropic's API. It lets developers use C… | 29 | 1656 | active |
| weijunext/smart-excel-ai SmartExcel is a web application that generates Excel formulas from natural-language descriptions using the ChatGPT API with streaming respo… | 37 | 1655 | active |
| huggingface/aisheets Hugging Face AI Sheets is an open-source no-code web application for building, enriching, and transforming datasets using AI models. It can… | 64 | 1652 | active |
| Roy3838/Observer Observer AI is a desktop application for building micro-agents that observe screen, camera, microphone, and audio inputs, process them with… | 88 | 1651 | active |
| Jeffser/Alpaca Alpaca is a GTK4/GNOME desktop client for Ollama that lets users manage, download, and chat with multiple local AI models, with support for… | 83 | 1651 | active |
| badlogic/lemmy A TypeScript monorepo providing a unified LLM interface library across Anthropic, OpenAI, and Google providers, plus a terminal UI framewor… | 32 | 1646 | active |
| HumanSignal/Adala Adala is a Python framework for building autonomous data labeling agents powered by LLMs. Agents learn skills iteratively from a ground tru… | 63 | 1644 | active |
| vigorX777/ai-daily-digest A zero-dependency TypeScript CLI tool (runnable as an OpenCode skill) that fetches the latest posts from 90 top tech blogs recommended by A… | 42 | 1643 | active |
| pengxiao-song/LaWGPT LaWGPT is a series of open-source large language models tuned with Chinese legal knowledge, built on Chinese-LLaMA/ChatGLM base models with… | 30 | 6061 | maintenance |
| AK391/ai-gradio A Python package built on Gradio that provides a unified interface for building AI apps with 15+ providers like OpenAI, Anthropic, and Gemi… | 25 | 1640 | active |
| SevaSk/ecoute Ecoute is a live transcription application that captures audio from both the user's microphone and speakers and displays real-time transcri… | 61 | 6050 | maintenance |
| nasa-jpl/rosa ROSA is a Python library and AI agent built on LangChain that lets developers interact with ROS1 and ROS2 robotics systems using natural la… | 56 | 1638 | active |
| Pickle-Pixel/ApplyPilot ApplyPilot is an open-source AI agent that autonomously applies to jobs on your behalf across any job site or application form. It runs a 6… | 54 | 1638 | active |
| ShengranHu/ADAS ADAS (Automated Design of Agentic Systems) is the official research code for the ICLR 2025 paper introducing Meta Agent Search, where a met… | 24 | 1638 | active |
| mlcommons/inference Reference implementations of the MLPerf Inference benchmark suite, the industry-standard benchmarks for measuring how fast ML systems run m… | 79 | 1631 | active |
| traycerai/traycer Traycer is an open-source desktop AI orchestration app that lets developers connect existing coding agents (Claude Code, Codex, Cursor, etc… | 88 | 1628 | active |
| google-gemma/gemma-translator An on-device, fully offline voice translator application powered by Gemma 4 via LiteRT-LM, with a React web frontend optimized for small ha… | 57 | 1628 | active |
| andrew-waters/orchard Orchard is a native Swift macOS GUI for Apple's container tooling, serving as a Docker Desktop replacement for managing containers, Linux V… | 85 | 1627 | active |
| MurrayHill1127/reopencode ReOpenCode is a Rust-based AI coding assistant providing a full agent loop with 18 callable tools, a terminal TUI (ratatui), and an HTTP RE… | 51 | 1627 | active |
| Meta-Harness Meta-Harness is a Python framework for automated end-to-end search over task-specific model harnesses — the code surrounding a fixed base L… | 57 | 1625 | active |
| Norsico/Video-Materials-AutoGEN-Workstation A self-hosted short-video production workstation that combines AI script generation (Gemini), batch TTS voiceover, AI image asset synthesis… | 62 | 1624 | active |
| a16z-infra/companion-app A tutorial starter stack for building and hosting AI companions with personality, backstory, and conversational memory. It combines Next.js… | 30 | 5981 | maintenance |
| modem-works/dream-recorder Dream Recorder is an open-source DIY bedside device built on a Raspberry Pi 5 that records users speaking their dreams aloud and turns them… | 38 | 1622 | active |
| ggml-org/Llama-macOS A native macOS menu bar app for running local LLMs, built on llama.cpp by the GGML org. It exposes an OpenAI-compatible local server at loc… | 86 | 1621 | active |
| clusterzx/paperless-ai A self-hosted web application that automatically analyzes, tags, and classifies documents in Paperless-ngx using LLMs via OpenAI-compatible… | 70 | 5972 | maintenance |
| AI Horde AI Horde is a crowdsourced distributed cluster where volunteers share GPU/CPU compute to generate AI images and text for others for free. T… | 77 | 1620 | active |
| NVIDIA/RULER RULER is a benchmark framework from NVIDIA that generates synthetic test examples to measure the true effective context size of long-contex… | 64 | 1619 | active |
| waybarrios/vllm-mlx vllm-mlx is a vLLM-style LLM inference server for Apple Silicon Macs built on native MLX, exposing both OpenAI and Anthropic compatible API… | 82 | 1618 | active |
| UbiquitousLearning/mllm MLLM is a fast, lightweight multimodal LLM inference engine written in C++ for mobile and edge devices, with backends for ARM CPU, Qualcomm… | 71 | 1617 | active |
| nilsherzig/LLocalSearch LLocalSearch is a self-hosted search aggregator that uses locally running LLM agents to answer questions with web-sourced information, show… | 10 | 5947 | maintenance |
| Soprano Soprano is an ultra-lightweight 80M-parameter text-to-speech model and Python library for fast, expressive, high-fidelity speech synthesis … | 41 | 1614 | active |
| agentlas-ai/Agentlas-OS Agentlas OS (with the Hephaestus engine) is an open-source agent framework that lets users create portable AI agents from plain-language re… | 78 | 1613 | active |
| langchain-ai/deepagentsjs Deep Agents (deepagentsjs) is a TypeScript library from LangChain for building controllable LLM-powered agents on top of LangGraph. It prov… | 82 | 1611 | active |
| MoonshotAI/Kimi-Linear Kimi Linear is a hybrid linear attention architecture (Kimi Delta Attention, based on Gated DeltaNet) released by Moonshot AI with 48B-para… | 37 | 1607 | active |
| hhhuang/CAG Cache-Augmented Generation (CAG) is a Python implementation of a retrieval-free alternative to RAG that preloads knowledge into an LLM's co… | 29 | 1607 | active |
| Tencent-Hunyuan/HY-WorldPlay HY-WorldPlay (HY-World 1.5) is Tencent Hunyuan's open-source framework for interactive 3D world modeling, generating explorable 3D scenes f… | 52 | 1604 | active |
| haibbo/cf-openai-azure-proxy A Cloudflare Worker script that proxies OpenAI-compatible API requests to Azure OpenAI Service, letting standard OpenAI clients use Azure d… | 21 | 1604 | active |
| THUDM/WebGLM WebGLM is an efficient web-enhanced question answering system (KDD 2023) that combines a large language model with web search retrieval and… | 32 | 1602 | active |
| enoch3712/ExtractThinker ExtractThinker is a Python document intelligence library that uses LLMs to extract and classify structured data from documents like PDFs, i… | 71 | 1601 | active |
| xLLM-AI/xllm xLLM is a high-performance C++ inference engine for LLM, VLM, DiT and recommendation models, optimized for heterogeneous AI accelerators su… | 80 | 1600 | active |
| Arboris Novel Arboris-Novel is a self-hosted AI writing companion web application for fiction authors, built in Python and deployable via Docker. It help… | 49 | 1600 | active |
| ponysb/91Writing A browser-based AI novel-writing application built with Vue 3 and Element Plus that integrates user-configured LLM APIs (GPT, Claude, Gemin… | 36 | 1600 | active |
| Hyperbrowser HyperAgent is a TypeScript library and CLI that adds LLM-powered natural language commands to Playwright for browser automation. It support… | 47 | 1598 | active |
| svnscha/mcp-windbg A Model Context Protocol (MCP) server that bridges AI models with Windows debuggers (CDB and KD) for crash dump analysis, user-mode remote … | 85 | 1597 | active |
| pytorch/FBGEMM FBGEMM is a collection of highly optimized low-precision matrix multiplication and convolution kernels for server-side deep learning infere… | 91 | 1595 | active |
| RamiAwar/dataline DataLine is an open-source, privacy-focused AI data analysis and visualization application that lets users chat with their data in natural … | 54 | 1594 | active |
| qnguyen3/chat-with-mlx A chat playground application for running LLMs locally on Apple Silicon Macs using Apple's MLX framework. It provides a chat UI with suppor… | 26 | 1594 | active |
| BAI-LAB/MemoryOS MemoryOS is a Python library that provides an operating-system-inspired memory layer for personalized AI agents, using a hierarchical short… | 59 | 1593 | active |
| byjlw/video-analyzer A Python CLI tool that analyzes videos by extracting key frames, transcribing audio with Whisper, and describing content using vision LLMs … | 54 | 1593 | active |
| NousResearch/autonovel An autonomous AI agent pipeline that writes, revises, evaluates, typesets, illustrates, and narrates a complete novel from a seed concept. … | 44 | 1590 | active |
| m1guelpf/plz-cli plz is a Rust-based CLI tool that acts as 'Copilot for your terminal', generating bash shell scripts from natural language descriptions usi… | 23 | 1590 | active |
| amd/gaia GAIA is AMD's open-source Python framework for building AI agents that run entirely locally on AMD Ryzen AI hardware, using NPU and iGPU ac… | 90 | 1587 | active |
| axoviq-ai/synthadoc Synthadoc is an open-source, domain-agnostic LLM knowledge compilation engine that transforms raw documents into structured, human-readable… | 82 | 1587 | active |
| cyberark/FuzzyAI FuzzyAI is an automated LLM fuzzing tool from CyberArk that tests LLM APIs for jailbreak vulnerabilities. It ships as a Python CLI with a w… | 48 | 1587 | active |
| D-Star-AI/dsRAG dsRAG is a high-performance retrieval engine for unstructured data, implemented as a Python library for building RAG pipelines. It improves… | 45 | 1586 | active |
| RWKV/rwkv.cpp A C++ port of the RWKV language model to the ggml tensor library, providing FP32, FP16, and quantized INT4/INT5/INT8 inference focused on C… | 34 | 1583 | active |
| LangGraph A Python library for building swarm-style multi-agent systems on top of LangGraph, where specialized agents dynamically hand off control to… | 70 | 1580 | active |
| turnstonelabs/turnstone Turnstone is a self-hosted, local-first orchestration platform for tool-using AI agents, letting LLMs use real tools like shell, files, sea… | 78 | 1578 | active |
| patched-codes/patchwork Patchwork is an open-source agentic AI framework and CLI that automates development chores like PR reviews, bug fixing, security patching, … | 68 | 1577 | active |
| evo-design/evo Evo is a 7-billion-parameter DNA foundation model built on the StripedHyena architecture, trained on ~300 billion tokens of prokaryotic who… | 65 | 1576 | active |
| Memento-Teams/Memento-Skills Memento-Skills is a generalist, continually-learnable LLM agent framework where reusable skills stored as markdown files serve as persisten… | 71 | 1575 | active |
| hexdocom/lemonai Lemon AI is a full-stack open-source general AI agent platform that runs locally, offering an alternative to Manus and Genspark AI. It comb… | 50 | 1572 | active |
| MLSysOps/MLE-agent MLE-Agent is an LLM-powered CLI agent that acts as a pairing assistant for machine learning engineers and researchers. It automates ML task… | 53 | 1570 | active |
| RunAnywhere RCLI is a single-binary CLI that runs open-source AI models locally on your machine, covering chat, vision, speech-to-text, text-to-speech,… | 82 | 1566 | active |
| K-Dense-AI/karpathy Karpathy is an open-source agentic Machine Learning Engineer that trains state-of-the-art ML models using the Claude Agent SDK and Google A… | 58 | 1566 | active |
| unitedbyai/droidclaw DroidClaw is an AI agent that controls Android phones via ADB, using a perceive-reason-act loop over accessibility trees and screenshots to… | 54 | 1566 | active |
| QuantaAlpha/QuantaAlpha QuantaAlpha is an LLM-driven framework for mining quantitative alpha factors using a trajectory-based self-evolving paradigm. Users describ… | 52 | 1563 | active |
| baaivision/Emu3.5 Emu3.5 is BAAI's native multimodal foundation model that jointly predicts next states across vision and language, trained on 10T+ interleav… | 40 | 1561 | active |
| neovateai/neovate-code Neovate Code is an open-source AI coding agent that runs in the terminal to generate code, fix bugs, review code, and add tests. It support… | 56 | 1557 | active |
| openai/openai-java The official Java SDK for accessing the OpenAI REST API, written in Kotlin and published to Maven Central. It provides typed clients for th… | 86 | 1554 | active |
| afc163/fanyi Fanyi is a command-line Chinese-English translation tool that aggregates results from iciba, youdao, and an LLM (with a free built-in proxy… | 79 | 1554 | active |
| QwenAudio/Fun-ASR Fun-ASR is a family of open-source LLM-based end-to-end speech recognition models from Tongyi Lab, covering Chinese, dialects, accents, and… | 79 | 1553 | active |
| AIDE ML AIDE ML is an open-source Python package implementing an LLM-driven agent that uses agentic tree search to autonomously write, debug, and i… | 60 | 1553 | active |
| Bklieger/infinite-bookshelf Infinite Bookshelf is a Streamlit web app that generates entire nonfiction books from a one-line prompt using Llama models on Groq. It scaf… | 32 | 1551 | active |
| amicalhq/amical Amical is an open-source, local-first AI dictation app for macOS and Windows that transcribes speech with Whisper and enhances it with LLMs… | 84 | 1548 | active |
| RayBytes/ChatMock ChatMock is a Python tool that exposes OpenAI- and Ollama-compatible API endpoints backed by a ChatGPT/Codex account login. It runs a local… | 65 | 1548 | active |