Open-source web-scraping software
2000 open-source libraries, tools and applications tagged web-scraping, ranked by how closely web-scraping is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
2000 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| tls-client A Go HTTP client library with a net/http-like interface that lets you select specific browser TLS fingerprints (Chrome, Firefox, Safari, et… | 90 | 1879 | active |
| alvarcarto/url-to-pdf-api A self-hosted microservice that converts URLs or raw HTML into PDF or PNG/JPEG images using Headless Chrome via Puppeteer. It exposes a sim… | 32 | 7104 | maintenance |
| zhu1090093659/deepseek-pp DeepSeek++ is an open-source browser extension for Chrome, Edge, and Firefox that turns the DeepSeek web app into an AI agent workspace. It… | 10 | 1870 | active |
| DeppWang/youdaonote-pull A Python script that exports and backs up all notes from Youdao Note (有道云笔记) to local storage. It converts note files to Markdown and downl… | 23 | 1866 | active |
| RanFeng/clipsketch-ai ClipSketch AI is a web-based AI content creation workbench that imports videos from Bilibili and Xiaohongshu links, lets users frame-accura… | 38 | 1858 | active |
| iojw/socialscan socialscan is a Python library and CLI tool that checks whether usernames and email addresses are taken, available, or invalid across onlin… | 62 | 1856 | active |
| j3ssie/metabigor Metabigor is a Go-based command-line OSINT tool that maps a target's infrastructure—IP ranges, subdomains, related domains, open ports, and… | 80 | 1853 | active |
| diskoverdata/diskover-community Diskover Community Edition is an open-source file system indexer and search engine that crawls local, NFS/SMB, and cloud storage and indexe… | 67 | 1823 | active |
| whotto/Video_note_generator A Python tool that converts video URLs into polished Xiaohongshu (Little Red Book) notes and blog articles. It downloads videos, transcribe… | 40 | 1819 | active |
| jacoblee93/fully-local-pdf-chatbot A Next.js web application that lets users chat with uploaded PDF documents entirely locally, performing chunking, embedding, vector storage… | 49 | 1817 | active |
| Planetable/Planet Planet is a free, open-source native macOS app (SwiftUI) for building and hosting decentralized blogs and websites on IPFS, with a built-in… | 92 | 1814 | active |
| AI Insight Daily A Cloudflare Workers-based content aggregation and generation platform that daily curates AI industry news, trending open-source projects, … | 64 | 1809 | active |
| iBUHub/AIStudioToAPI A self-hosted proxy service that wraps the Google AI Studio Build web interface into OpenAI, Gemini, and Anthropic compatible APIs using br… | 83 | 1789 | active |
| arabold/docs-mcp-server A self-hosted MCP server that indexes documentation from websites, GitHub, npm, PyPI, and local files so AI coding assistants can query up-… | 82 | 1786 | active |
| br-g/openf1 OpenF1 is a free, open-source API providing real-time and historical Formula 1 data, including lap timings, car telemetry, driver info, and… | 62 | 1779 | active |
| vexorian/dizquetv dizqueTV is a self-hosted web application that creates simulated live TV channels from media on your Plex servers. It exposes channels via … | 82 | 1768 | active |
| xmendez/wfuzz Wfuzz is a Python-based command-line web application fuzzer that replaces a FUZZ keyword in HTTP requests with values from configurable pay… | 54 | 6588 | maintenance |
| maxbbraun/trump2cash A Python stock trading bot that watches Donald Trump's tweets, detects mentions of publicly traded companies, and uses sentiment analysis t… | 10 | 6545 | maintenance |
| kyleskom/NBA-Machine-Learning-Sports-Betting A Python application that predicts NBA game winners and totals (over/under) using XGBoost and neural network models trained on historical t… | 78 | 1747 | active |
| Freeboard/freeboard Freeboard is an open-source, browser-based real-time dashboard builder and viewer for IoT and web mashups, serving as a free alternative to… | 32 | 6506 | maintenance |
| glidea/zenfeed zenfeed is a self-hosted AI-powered RSS reader and information hub built in Go. It aggregates RSS feeds, uses LLMs to filter, summarize, an… | 62 | 1716 | active |
| MorDavid/BruteForceAI BruteForceAI is a Python-based penetration testing tool that uses LLMs (via Ollama or Groq) to automatically analyze login page HTML and id… | 56 | 1715 | active |
| FroggMaster/CreamInstaller A Windows GUI application that automatically finds installed Steam, Epic, and Ubisoft games, parses SteamCMD and store APIs for DLC informa… | 89 | 1712 | active |
| fish2018/webhtv WebHomeTV is an Android video streaming app (mobile and TV) forked from the FongMi/CatVod ecosystem, adding a customizable web-based homepa… | 80 | 1712 | active |
| ly525/luban-h5 Luban H5 is a self-hosted visual page builder for creating mobile H5 pages via drag-and-drop, similar to Webflow or Chinese platforms like … | 46 | 6241 | maintenance |
| WangYihang/GitHacker GitHacker is a multi-threaded Python CLI tool that exploits exposed `.git` directories on web servers to reconstruct the entire Git reposit… | 75 | 1673 | active |
| TurboSeek TurboSeek is an open-source AI search engine web application inspired by Perplexity, built with Next.js and TypeScript. It searches the web… | 62 | 1661 | active |
| goldvideo H.265 Web Player A complete web-based H.265 (HEVC) video player solution built in JavaScript. It uses JS-based stream demuxing, WebAssembly (FFmpeg) video d… | 64 | 1660 | active |
| Munter/subfont A Node.js command line tool that statically analyzes HTML pages to generate optimal webfont subsets based on the characters actually used. … | 66 | 1638 | active |
| siddsachar/row-bot Row-Bot is a local-first desktop AI assistant and workbench that combines chat, durable memory, a personal knowledge graph, tool use, paren… | 83 | 1637 | active |
| Simpleyyt/ai-manus AI Manus is a self-hostable, general-purpose AI Agent system with a WebUI that plans and executes tasks using tools like Terminal, Browser,… | 84 | 1629 | active |
| web-arena-x/webarena WebArena is a standalone, self-hostable web environment and benchmark for building and evaluating autonomous web agents. It provides realis… | 40 | 1620 | active |
| nilsherzig/LLocalSearch LLocalSearch is a self-hosted search aggregator that uses locally running LLM agents to answer questions with web-sourced information, show… | 10 | 5947 | maintenance |
| karpathy/arxiv-sanity-preserver A self-hostable web application for browsing, searching, and filtering recent arXiv submissions, with personalized paper recommendations ba… | 32 | 5799 | maintenance |
| hexdocom/lemonai Lemon AI is a full-stack open-source general AI agent platform that runs locally, offering an alternative to Manus and Genspark AI. It comb… | 50 | 1572 | active |
| robocorp/rpaframework RPA Framework is a collection of open-source Python libraries and tools for Robotic Process Automation, usable from both Robot Framework an… | 94 | 1570 | active |
| microsoft/playwright-java Playwright for Java is a library to automate Chromium, Firefox, and WebKit browsers with a single API, built for reliable end-to-end testin… | 96 | 1562 | active |
| google/neuroglancer Neuroglancer is a WebGL-based, client-side web application for visualizing large 3-D volumetric datasets, supporting arbitrary cross-sectio… | 77 | 1562 | active |
| Prismer-AI/PrismerCloud Prismer Cloud is a hosted/self-hostable knowledge infrastructure platform for AI agents, providing context fetching and compression, docume… | 68 | 1558 | active |
| willmiao/ComfyUI-Lora-Manager A ComfyUI extension that provides a web-based manager for organizing, previewing, downloading, and applying LoRA models with metadata and r… | 85 | 1543 | active |
| RTGS2017/NagaAgent NagaAgent is a desktop AI personal assistant framework with an anime (Live2D) virtual character, built in Python with an Electron front end… | 78 | 1542 | active |
| helloxz/zdir Zdir is a self-hosted private storage and file-sharing application built with Golang and Vue3 that indexes directories, previews files onli… | 23 | 1536 | active |
| emcf/thepipe thepipe is a Python package and API that extracts clean markdown, multimodal media, and structured data from tricky documents like PDFs, Wo… | 79 | 1529 | active |
| import-ai/omnibox OmniBox is a cross-platform, self-hostable AI knowledge hub that lets users collect webpages, files, and voice notes, then parse, index, an… | 89 | 1527 | active |
| memfreeme/memfree MemFree is an open-source hybrid AI search engine that answers questions using both your personal knowledge base and the web, integrating m… | 62 | 1515 | active |
| DemonDamon/FinnewsHunter FinnewsHunter is a multi-agent financial intelligence platform built on the AgenticX framework that monitors Chinese financial news sources… | 76 | 1500 | active |
| ViewTube/viewtube ViewTube is an open-source, privacy-conscious alternative frontend for YouTube that lets users watch, search, and discover videos without a… | 66 | 1476 | active |
| microlinkhq/unavatar unavatar.io is a unified avatar API that retrieves user profile images from a single URL by username, email, or domain across 72 platforms … | 96 | 1473 | active |
| FACT (Firmware Analysis and Comparison Tool) FACT (Firmware Analysis and Comparison Tool) is a self-hosted Python application that automates firmware security analysis for devices like… | 94 | 1468 | active |
| placemark/placemark Placemark is a free, open-source web application for creating, viewing, editing, and analyzing geospatial map data, built with MapLibre GL … | 82 | 1462 | active |
| ThoughtfulDev/EagleEye EagleEye is a Python-based OSINT tool that identifies social media profiles (Instagram, Facebook, Twitter, YouTube) of a person using face … | 32 | 5231 | maintenance |
| t3l3machus/toxssin toxssin is an open-source penetration testing CLI tool that automates exploitation of Cross-Site Scripting (XSS) vulnerabilities. It pairs … | 32 | 1443 | active |
| hteen/apple-store-helper A cross-platform desktop GUI application written in Go (using the Fyne framework) that monitors Apple Store inventory for specific iPhone m… | 50 | 1442 | active |
| eumemic/ai-legion AI Legion is a TypeScript framework for running LLM-powered autonomous agents that collaborate to accomplish tasks. Agents are managed thro… | 37 | 1437 | active |
| caarlos0/starcharts A Go web service that generates SVG charts plotting a GitHub repository's stars over time, served as embeddable images via starchart.cc. It… | 75 | 1425 | active |
| millylee/anyrouter-check-in A Python tool that automates daily check-ins on LLM API relay platforms like Any Router and Agent Router, theoretically compatible with any… | 66 | 1424 | active |
| GauravSingh9356/J.A.R.V.I.S A Python-based voice-controlled personal assistant inspired by Iron Man's J.A.R.V.I.S. It combines speech recognition, text-to-speech, OCR,… | 44 | 1415 | active |
| zubair-trabzada/ai-sales-team-claude A collection of 14 Claude Code skills and 5 parallel agents that turn Claude Code into an AI-powered sales team. It researches prospects, q… | 45 | 1404 | active |
| McGill-NLP/webllama WebLlama is a framework for building Llama-3-powered agents that browse the web by following natural language instructions and dialogue. It… | 56 | 1399 | active |
| gerbera/gerbera Gerbera is a free, open-source UPnP/DLNA media server that streams digital media across a home network to compatible devices like TVs, game… | 94 | 1396 | active |
| ChatWise ChatWise is a fast, privacy-friendly desktop AI chatbot client that connects to multiple LLM providers (OpenAI, Anthropic, Google, Groq, Ol… | 67 | 1371 | active |
| mishl-dev/Discord-AI-Chatbot A self-hosted Discord AI chatbot application written in Python that uses the Groq API for fast LLM responses. It supports slash and text co… | 10 | 1363 | active |
| sashiko-dev/sashiko Sashiko is an agentic code review system for the Linux kernel that ingests patches from mailing lists, GitHub PRs, GitLab MRs, or local git… | 61 | 1346 | active |
| 1692775560/deepseek_project A collection of DeepSeek-related tools including a WeChat chatbot that uses the DeepSeek API for automated replies, a local document upload… | 63 | 1341 | active |
| Emupedia/emupedia.github.io Emupedia is a nonprofit web-based platform that archives and preserves retro video games and software, making them playable in modern brows… | 72 | 1338 | active |
| insoxin/API A free, self-hostable REST API platform (姬长信 API) packaged as a Docker image, offering dozens of endpoints for daily-life services, news, w… | 65 | 1336 | active |
| moshstudio/TAICHI-flet TAICHI-flet is a Windows desktop entertainment application built with the Flet framework that lets users browse images, music, novels, comi… | 74 | 4736 | maintenance |
| benjaminjonard/koillection Koillection is a self-hosted web application for managing collections of any kind, such as books, DVDs, stamps, and games. It supports coll… | 96 | 1328 | active |
| shift-labs-ai/markit A Rust-based tool that converts documents, data files, web pages, and media into Markdown, usable both as a CLI and a library. It supports … | 75 | 1325 | active |
| ksimback/hermes-ecosystem Hermes Atlas is a community-curated web directory mapping the ecosystem of tools, skills, plugins, and integrations built around Hermes Age… | 60 | 1323 | active |
| Marven11/Fenjing Fenjing is an automated Jinja2 SSTI (server-side template injection) exploitation tool designed for CTF competitions. It automatically anal… | 85 | 1310 | active |
| camoufox-reverse-mcp An AI-powered 'Skill' package for JavaScript reverse engineering that plugs into AI coding tools like Claude Code, Cursor, and Codex. It pr… | 78 | 1308 | active |
| 0xInfection/XSRFProbe XSRFProbe is a Python-based Cross Site Request Forgery (CSRF/XSRF) audit and exploitation toolkit. It crawls web applications, runs systema… | 76 | 1306 | stable |
| okfn-brasil/serenata-de-amor Operação Serenata de Amor is an open-source data science project that uses machine learning to audit Brazilian congresspeople's public expe… | 32 | 4612 | maintenance |
| huangkiki/dailypaper-skills A set of Claude Code skills that automates a daily research paper pipeline: fetching new papers from HuggingFace Daily, Trending, and arXiv… | 60 | 1301 | active |
| lardbit/nefarious Nefarious is a self-hosted web application that automatically searches for and downloads movies and TV shows via torrents, using Jackett fo… | 77 | 1299 | active |
| codefuse-ai/codefuse-chatbot CodeFuse-ChatBot is an open-source AI assistant for the software development lifecycle, combining a multi-agent scheduling framework with D… | 28 | 1295 | active |
| yzfly/douyin-mcp-server A Python MCP server and WebUI that extracts watermark-free Douyin (TikTok China) video download links and transcribes video speech into tex… | 10 | 1273 | active |
| franklioxygen/MyTube MyTube is a self-hosted web application that downloads videos from YouTube, Bilibili, Twitch, MissAV, and any yt-dlp-supported site, storin… | 65 | 1251 | active |
| xllm-go/bypass A Go server that reverse-engineers the chat interfaces of multiple AI providers (Coze, DeepSeek, Cursor, Windsurf, Grok, Bing Copilot, You,… | 56 | 1245 | active |
| rohunvora/x-research-skill A TypeScript CLI and agent skill that wraps the X/Twitter API for searching tweets, pulling threads, monitoring accounts, and generating so… | 43 | 1241 | active |
| 666ghj/DeepSearchAgent-Demo A framework-free Python implementation of a deep search AI agent that generates high-quality research reports through multi-round web searc… | 31 | 1229 | active |
| Unrud/video-downloader A GTK desktop application for downloading videos from websites like YouTube, built on top of yt-dlp. It offers an easy-to-use GUI with feat… | 89 | 1226 | active |
| LifeArchiveProject/BilibiliHistoryFetcher A Python/FastAPI backend tool that fetches, stores, and analyzes a user's Bilibili watch history, favorites, dynamics, comments, and intera… | 76 | 1223 | active |
| aydinnyunus/ai-captcha-bypass A Python command-line tool that uses multimodal LLMs (GPT-4o, Gemini) to automatically solve various CAPTCHA types, including text, reCAPTC… | 58 | 1219 | active |
| ZHangZHengEric/Sage Sage is a Python-based multi-agent framework and platform for planning, executing, and delivering complex tasks with LLM-powered agents. It… | 87 | 1214 | active |
| Open Semantic Search An open-source integrated search server and ETL framework for processing, analyzing, and exploring large document collections. It combines … | 36 | 1213 | active |
| gys619/Thread A personal collection of automation scripts for the Qinglong task scheduler, QuantumultX, QQ bots, and shell environments, written in JavaS… | 67 | 1210 | active |
| zas023/JdBuyer JdBuyer is a Python-based desktop application that monitors JD.com (Jingdong) product stock availability and automatically attempts to plac… | 23 | 4226 | maintenance |
| Hackmanit/Web-Cache-Vulnerability-Scanner A fast, Go-based CLI scanner for detecting web cache poisoning and web cache deception vulnerabilities. It supports many attack techniques,… | 55 | 1207 | active |
| makiisthenes/TiktokAutoUploader A Python CLI tool that automatically edits and uploads videos to TikTok using HTTP requests instead of browser automation, making uploads f… | 100 | 1199 | active |
| ALwrity/ALwrity ALwrity is an open-source, AI-first digital marketing platform built in Python that combines content strategy planning, multimodal AI conte… | 77 | 1186 | active |
| thinkany-ai/rag-search A RAG (retrieval-augmented generation) search API service built with FastAPI by the ThinkAny AI search engine team. It combines web search … | 26 | 1180 | active |
| jenish-sojitra/JSAnalyzer A Burp Suite extension written in Python (Jython) that performs static analysis on JavaScript files proxied through Burp. It extracts API e… | 41 | 1175 | active |
| test-zeus-ai/testzeus-hercules Hercules is an open-source AI testing agent that turns Gherkin feature files into fully automated end-to-end tests covering UI, API, securi… | 80 | 1172 | active |
| vm0-ai/vm0 VM0 (Zero) is an AI agent platform that acts as a 'trustworthy AI teammate', connecting to 1,400+ tools (Slack, Notion, Gmail, Linear, GitH… | 81 | 1161 | active |
| Libr-AI/OpenFactVerification Loki is an open-source Python tool that automates fact verification by decomposing texts into claims, retrieving evidence via search, and u… | 16 | 1159 | active |
| jkwill87/mnamer mnamer is a Python command-line utility that parses media filenames for metadata, queries online providers like TVDb, TvMaze, TMDb, and OMD… | 91 | 1156 | active |
| tholian-network/stealth Stealth is a secure, privacy-focused web browser, scraper, and proxy built in JavaScript that emphasizes automation, bandwidth efficiency, … | 32 | 1149 | active |
| GamehunterKaan/AutoPWN-Suite AutoPWN Suite is a Python-based automated vulnerability scanning and exploitation framework that wraps nmap for host discovery, version-bas… | 100 | 1142 | active |