Open-source llm-inference software
3358 open-source libraries, tools and applications tagged llm-inference, ranked by how closely llm-inference is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
3358 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| Vaibhavs10/insanely-fast-whisper A CLI tool for transcribing audio files on-device using OpenAI's Whisper models, powered by Hugging Face Transformers, Optimum, and Flash A… | 46 | 13063 | active |
| zai-org/CogVideo CogVideo/CogVideoX is an open-source family of text-to-video and image-to-video generation models from Zhipu AI (THUDM), with inference and… | 42 | 13055 | active |
| txtai txtai is an all-in-one open-source AI framework for semantic search, LLM orchestration and language model workflows, built around an embedd… | 95 | 12978 | active |
| StarTrail-org/LEANN LEANN is a low-storage vector index that enables retrieval-augmented generation (RAG) on personal devices by recomputing embeddings on the … | 84 | 12966 | active |
| Open Deep Research An open-source deep research agent built with LangGraph that performs multi-step web research using configurable LLM providers, search tool… | 10 | 12680 | active |
| Wei-Shaw/claude-relay-service A self-hosted relay/proxy service for Claude Code (and OpenAI, Gemini, Droid) subscriptions that lets multiple users share accounts via ind… | 83 | 12667 | active |
| HKUDS/ViMax ViMax is an agentic video generation framework that uses coordinated multi-agent collaboration (director, screenwriter, producer, video gen… | 71 | 12666 | active |
| nearai/ironclaw IronClaw is an open-source, Rust-based 'Agent OS' and personal AI assistant that runs locally or on NEAR AI Cloud, with a WebUI, terminal, … | 79 | 12629 | active |
| Tencent-Hunyuan/HunyuanVideo HunyuanVideo is Tencent's open-source framework for large-scale video generation, providing PyTorch model definitions, pre-trained weights,… | 58 | 12577 | active |
| 0x4m4/hexstrike-ai HexStrike AI is an MCP server that bridges LLM agents (Claude, GPT, Copilot) with 150+ cybersecurity tools for autonomous penetration testi… | 58 | 12431 | active |
| ubicloud/ubicloud Ubicloud is an open-source IaaS cloud platform that provides elastic compute (VMs), block storage, networking, load balancing, managed Post… | 76 | 12316 | active |
| FlagOpen/FlagEmbedding FlagEmbedding is the official Python toolkit for BAAI's BGE family of embedding models and rerankers, covering inference, evaluation, and f… | 81 | 12242 | active |
| jina-ai/reader Jina AI Reader converts any URL into LLM-friendly markdown via the r.jina.ai prefix, and searches the web into markdown via s.jina.ai. It r… | 59 | 12158 | active |
| Tongyi-MAI/Z-Image Z-Image is a 6B-parameter text-to-image generation foundation model family built on a single-stream diffusion transformer, with a distilled… | 43 | 12057 | active |
| Bisheng BISHENG is an open-source LLM DevOps (LLMOps) platform for building enterprise AI applications, offering GenAI workflow orchestration, RAG,… | 92 | 12032 | active |
| MemTensor/MemOS MemOS is a memory operating system for LLMs and AI agents that unifies storing, retrieving, and managing long-term, multi-modal memory thro… | 85 | 12014 | active |
| H2O h2oGPT is an Apache-2.0 open-source application for chatting with local, private LLMs and querying/summarizing your own documents (PDFs, Wo… | 10 | 11942 | active |
| Scira Scira is an open-source, minimalistic AI-powered search engine and agentic research platform that plans multi-step searches, retrieves live… | 64 | 11904 | active |
| facebookresearch/seamless_communication A library of foundational multilingual multimodal AI models from Meta for speech and text translation, including SeamlessM4T, SeamlessExpre… | 71 | 11889 | active |
| ludwig-ai/ludwig Ludwig is a declarative, low-code deep learning framework for training, fine-tuning, and deploying AI models — from LLMs to tabular, image,… | 96 | 11775 | active |
| Pest Pest is an elegant PHP testing framework built on top of PHPUnit, offering a functional test API with expressive expect() assertions, datas… | 92 | 11724 | active |
| CocoIndex CocoIndex is an open-source incremental data framework for AI, with a Rust engine and a declarative Python API that keeps sources (files, S… | 83 | 11711 | active |
| UI-TARS UI-TARS is ByteDance's open-source multimodal AI agent stack, comprising Agent TARS (a CLI/Web UI multimodal agent that operates terminals,… | 46 | 11567 | active |
| kangarooking/cangjie-skill Cangjie Skill is a Python-based tool that distills methodologies from high-value content like books, long-form videos, and podcasts into ex… | 78 | 11455 | active |
| LaurieWired/GhidraMCP GhidraMCP is a Model Context Protocol server implemented as a Ghidra plugin that exposes Ghidra's reverse engineering tools to LLM clients.… | 87 | 11418 | active |
| salesforce/LAVIS LAVIS is a Python library from Salesforce AI Research providing a unified toolkit for language-vision (multimodal) intelligence, including … | 10 | 11264 | active |
| Pocket Flow Pocket Flow is a minimalist 100-line LLM framework in Python with zero dependencies, modeling LLM applications as a graph of nodes connecte… | 51 | 11208 | active |
| TEN-framework/ten-framework TEN is an open-source framework for building real-time multimodal conversational AI agents, with a focus on low-latency voice assistants. I… | 87 | 11173 | active |
| LTX-Video Official repository for LTX-Video, a DiT-based open-weights video generation model from Lightricks that generates high-fidelity video (up t… | 46 | 11138 | active |
| bytebot-ai/bytebot Bytebot is a self-hosted, open-source AI desktop agent that operates a full containerized Linux desktop, using screen vision, mouse, and ke… | 10 | 11077 | active |
| ValueCell-ai/valuecell ValueCell is a community-driven, multi-agent platform for financial applications, providing AI investment agents for stock selection, resea… | 54 | 11031 | active |
| OpenVINO OpenVINO is an open-source toolkit from Intel for optimizing and deploying deep learning inference across CPU, GPU, and NPU hardware. It su… | 94 | 10996 | stable |
| Mail-0/Zero Zero (Mail0) is an open-source, self-hostable email client that unifies multiple providers like Gmail and Outlook into one privacy-first in… | 46 | 10857 | active |
| VoltAgent/voltagent VoltAgent is an open-source TypeScript framework for building production-ready AI agents with memory, tools, RAG, guardrails, MCP, voice, a… | 82 | 10773 | active |
| ChatGPTBox-dev/chatGPTBox A browser extension that deeply integrates ChatGPT into your browsing experience across Chrome, Edge, Firefox, Safari, and Android. It prov… | 91 | 10757 | active |
| thesysdev/openui OpenUI is an open-source, renderer-agnostic Generative UI framework built around OpenUI Lang, a compact streaming-first DSL that lets LLM a… | 84 | 10732 | active |
| facebookresearch/xformers xFormers is a PyTorch-based library of hackable, optimized Transformer building blocks with custom CUDA kernels for fast, memory-efficient … | 86 | 10565 | active |
| sigoden/aichat AIChat is an all-in-one command-line LLM client written in Rust that connects to 20+ providers including OpenAI, Claude, Gemini, and Ollama… | 60 | 10471 | active |
| niedev/RTranslator RTranslator is a free, open-source, offline real-time translation app for Android that runs speech recognition (Whisper) and translation (M… | 77 | 10449 | active |
| Kuberwastaken/claurst Claurst is an open-source, multi-provider terminal coding agent written in Rust, originally a clean-room reimplementation of Claude Code. I… | 82 | 10325 | active |
| StarTrail-org/PixelRAG PixelRAG is a Python library and hosted service for visual retrieval-augmented generation: it renders web pages and documents into screensh… | 77 | 10320 | active |
| snakers4/silero-vad Silero VAD is a pre-trained, enterprise-grade Voice Activity Detector model available via PyPI, runnable with PyTorch or ONNX Runtime. It d… | 93 | 10317 | stable |
| OthersideAI/self-operating-computer A Python framework that lets multimodal LLMs operate a computer by viewing the screen and issuing mouse and keyboard actions to reach a sta… | 45 | 10299 | active |
| Astrid Astrid is a portable, capability-secure operating system for composable software, where every component runs as a sealed WebAssembly capsul… | 81 | 10279 | active |
| microsoft/UFO UFO³ is Microsoft's open-source framework for LLM-powered GUI agents that automate Windows applications and orchestrate workflows across mu… | 93 | 10092 | active |
| companion-inc/feynman Feynman is an open-source AI research agent that reads papers, searches the web, writes cited research drafts, plans experiments, and ranks… | 79 | 10020 | active |
| lipku/LiveTalking LiveTalking is an open-source real-time interactive streaming digital human engine that drives a talking-head avatar from text or audio wit… | 88 | 9997 | active |
| kyutai-labs/pocket-tts Pocket TTS is a lightweight 100M-parameter text-to-speech model and Python library from Kyutai that runs efficiently on CPUs without a GPU.… | 83 | 9923 | active |
| OpenWhispr/openwhispr OpenWhispr is an open-source, privacy-first voice-to-text dictation desktop app for macOS, Windows, and Linux. It supports fully local tran… | 83 | 9595 | active |
| Higress Higress is a CNCF Sandbox, cloud-native AI-native API gateway built on Istio and Envoy, extensible with Wasm plugins written in Go, Rust, o… | 99 | 9568 | active |
| Claude Code Action A GitHub Action that runs Claude Code on pull requests and issues, answering questions, reviewing code, and implementing changes triggered … | 82 | 9434 | active |
| QwenAudio/SenseVoice SenseVoice is an open-source speech foundation model (SenseVoiceSmall) providing multilingual ASR, spoken language identification, speech e… | 87 | 9386 | active |
| langchain-ai/local-deep-researcher A fully local web research assistant that runs any LLM via Ollama or LMStudio to iteratively search the web, summarize findings, identify k… | 67 | 9378 | active |
| Kokoro TTS An inference library for Kokoro-82M, an open-weight text-to-speech model with 82 million parameters that delivers quality comparable to lar… | 34 | 9340 | active |
| NVIDIA/garak garak is a command-line LLM vulnerability scanner that probes large language models for failures like hallucination, data leakage, prompt i… | 89 | 9332 | active |
| CoAI.Dev (Chat Nio) CoAI.Dev is a self-hostable, multi-tenant AI platform combining a ChatGPT-style chat UI with an enterprise-grade unified LLM gateway suppor… | 50 | 9323 | active |
| Terax Terax is a lightweight (~7 MB) open-source desktop application that combines a GPU-rendered terminal, code editor, file explorer, git sourc… | 77 | 9305 | active |
| meituan-longcat/LongCat-Video LongCat-Video is a 13.6B-parameter foundational video generation model from Meituan that unifies text-to-video, image-to-video, and video-c… | 51 | 9230 | active |
| miurla/morphic Morphic is an open-source AI-powered search engine that generates grounded, cited answers with a generative UI that renders rich inline com… | 93 | 9190 | active |
| studio-dots-ai/dots.ocr dots.ocr is a 1.7B-parameter vision-language model for multilingual document layout parsing, converting documents into structured output wi… | 48 | 9157 | active |
| open-pencil/open-pencil OpenPencil is an open-source, AI-native design editor that opens and writes native Figma (.fig) files, with built-in AI chat that can creat… | 83 | 9153 | active |
| OpenSPG/KAG KAG (Knowledge Augmented Generation) is a Python framework built on the OpenSPG engine and large language models for logical form-guided re… | 46 | 9073 | active |
| maximhq/bifrost Bifrost is a high-performance, open-source AI gateway written in Go that unifies access to 23+ LLM providers (OpenAI, Anthropic, AWS Bedroc… | 84 | 9027 | active |
| jetson-inference A C++/Python DNN inference library and tutorial guide ('Hello AI World') for deploying deep learning vision models on NVIDIA Jetson devices… | 40 | 9015 | stable |
| Yearning Yearning is a self-hosted web-based SQL audit platform for MySQL, providing audit tickets with approval workflows, automated syntax checkin… | 75 | 8969 | active |
| AIClient2API AIClient2API is a self-hosted Node.js proxy that simulates client-only LLM service requests (Antigravity, Codex, Grok, Kiro, Gemini CLI, et… | 81 | 8896 | active |
| Builder.io GPT Crawler is a TypeScript‑based CLI that crawls websites, extracts content, and formats it into knowledge files for OpenAI custom GPTs. I… | 94 | 8862 | active |
| TeamWiseFlow/xiaobei Xiaobei is an open-source multi-agent system that automates social media marketing and customer acquisition for solo entrepreneurs and smal… | 89 | 8574 | active |
| lastmile-ai/mcp-agent mcp-agent is a Python framework for building AI agents on top of the Model Context Protocol (MCP) using simple, composable workflow pattern… | 48 | 8571 | active |
| LC044/WeChatMsg WeChatMsg (留痕/MemoTrace) is a desktop application built with PyQt that extracts, exports, and preserves WeChat chat history on a user's own… | 55 | 42101 | maintenance |
| QwenLM/Qwen-Image Qwen-Image is a 20B MMDiT image generation foundation model from the Qwen team, with strong complex text rendering (especially Chinese) and… | 45 | 8377 | active |
| zilliztech/deep-searcher DeepSearcher is an open-source deep research tool that combines LLMs (DeepSeek, OpenAI, Claude, Qwen, etc.) with vector databases like Milv… | 54 | 8292 | active |
| SuanmoSuanyangTechnology/MemoryBear MemoryBear is an AI memory management system that gives LLM agents and chatbots human-like long-term memory across a full lifecycle of perc… | 85 | 8286 | active |
| EstrellaXD/Auto_Bangumi AutoBangumi is a self-hosted, RSS-based automated anime downloading and organizing tool. It parses RSS feeds from sites like Mikan Project,… | 98 | 8285 | active |
| n4ze3m/page-assist Page Assist is an open-source browser extension that adds a sidebar and web UI for interacting with locally running AI models like Ollama f… | 94 | 8272 | active |
| cloudflare/agentic-inbox A self-hosted email client with an integrated AI agent that runs entirely on Cloudflare Workers. Incoming mail is handled via Cloudflare Em… | 47 | 8237 | active |
| Ucas-HaoranWei/GOT-OCR2.0 Official implementation of GOT-OCR2.0, a unified end-to-end vision-language model for general OCR that converts images of text, documents, … | 23 | 8228 | active |
| GetStream/Vision-Agents An open-source Python framework by Stream for building low-latency real-time voice and video AI agents. It provides 35+ provider plugins (O… | 83 | 8172 | active |
| Pinokio Pinokio is a desktop application that acts as a one-click launcher and browser for open-source AI apps, letting users discover, install, ru… | 91 | 8154 | active |
| MeiGen-AI/InfiniteTalk InfiniteTalk is an open-source model and framework for unlimited-length audio-driven talking video generation, supporting both image-to-vid… | 52 | 8129 | active |
| NullClaw NullClaw is an autonomous AI assistant/agent runtime written in Zig, distributed as a tiny static binary (~678 KB, ~1 MB RAM, <2 ms startup… | 69 | 8104 | active |
| leptonai/search_with_lepton A demo application for building a conversational AI search engine (Perplexity-style) in under 500 lines of code, combining LLMs with search… | 10 | 8054 | active |
| suno-ai/bark Bark is Suno's open-source transformer-based text-to-audio model that generates highly realistic multilingual speech, music, background noi… | 31 | 39276 | maintenance |
| MLX-Audio MLX-Audio is a Python library built on Apple's MLX framework for fast text-to-speech (TTS), speech-to-text (STT), and speech-to-speech (STS… | 88 | 8012 | active |
| R2R R2R is a production-ready AI retrieval system implementing agentic Retrieval-Augmented Generation (RAG) with a RESTful API. It provides sta… | 47 | 8010 | active |
| evidentlyai/evidently Evidently is an open-source Python framework for evaluating, testing, and monitoring ML and LLM-powered systems, with 100+ built-in metrics… | 97 | 7984 | active |
| logancyang/obsidian-copilot Copilot for Obsidian is an Obsidian community plugin that adds AI chat and agent capabilities to your note vault, integrating opencode, Cla… | 94 | 7874 | active |
| rikkahub/rikkahub RikkaHub is a native Android LLM chat client built with Kotlin and Jetpack Compose that supports switching between multiple AI providers su… | 81 | 7859 | active |
| TensorRTx A C++ collection of popular deep learning networks (YOLO variants, ResNet, MobileNet, Swin Transformer, OCR models, and more) implemented f… | 73 | 7838 | active |
| Azure Search OpenAI Demo A reference sample application implementing the Retrieval-Augmented Generation (RAG) pattern on Azure, combining Azure AI Search for docume… | 93 | 7776 | active |
| Flyte Flyte is an open-source workflow orchestration platform for coordinating data, ML models, and AI agents at scale, authored in pure Python a… | 96 | 7741 | active |
| zenstory-ai/oh-story-claudecode A skill pack for Claude Code and compatible AI coding agents that provides an end-to-end workflow for writing Chinese web fiction, covering… | 82 | 7700 | active |
| Osmantic/ODS ODS (Osmantic Deployment System) is a self-hosted AI server installer and runtime that turns a PC, Mac, or Linux machine into a private AI … | 82 | 7658 | active |
| SkyReels SkyReels-V2 is an open-source infinite-length film/video generative model using an AutoRegressive Diffusion-Forcing architecture, released … | 45 | 7589 | active |
| BiliNote BiliNote is an open-source AI video note-taking application that turns video links from Bilibili, YouTube, Douyin, and local files into str… | 81 | 7563 | active |
| apurvsinghgautam/robin Robin is an AI-powered dark web OSINT investigation tool that uses LLMs to refine queries, filter results from dark web search engines, and… | 86 | 7440 | active |
| LargeWorldModel/LWM Large World Model (LWM) is a family of open-source 7B-parameter multimodal autoregressive transformer models trained on long videos and boo… | 26 | 7425 | active |
| QuivrHQ/MegaParse MegaParse is a Python library that parses PDFs, Word, PowerPoint, Excel, CSV, and text documents into LLM-friendly formats with a focus on … | 26 | 7414 | active |
| zhinianboke/xianyu-auto-reply A self-hosted multi-account automation system for the Xianyu (Goofish) second-hand marketplace, built with Python/FastAPI, React, MySQL, Re… | 62 | 7399 | active |
| basketikun/infinite-canvas An open-source infinite canvas workbench for AI-driven visual creation, combining canvas orchestration, AI image generation, reference-imag… | 79 | 7369 | active |