Open-source web-scraping software
2000 open-source libraries, tools and applications tagged web-scraping, ranked by how closely web-scraping is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
2000 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| ginobefun/BestBlogs BestBlogs.dev is an AI-powered personal reading assistant and content curation platform that aggregates articles, podcasts, videos, tweets,… | 64 | 4053 | active |
| knownsec/aipyapp AiPy (Python-Use) is an AI desktop assistant from Knownsec that implements a 'Code is Agent' paradigm: it gives an LLM a full Python execut… | 57 | 4027 | active |
| leebaird/discover A collection of custom Bash and Python scripts that automate penetration testing tasks including reconnaissance, scanning, enumeration, and… | 77 | 3944 | active |
| remorses/playwriter Playwriter is a Chrome extension plus CLI/MCP tool that lets AI agents control your real, already-logged-in Chrome browser by executing Pla… | 81 | 3923 | active |
| jordan-gibbs/hyperresearch Hyperresearch is a Python CLI harness that turns Claude Code into a deep research agent running a 16-step adversarially-audited pipeline. I… | 81 | 3873 | active |
| gmail.js Gmail.js is a JavaScript library that provides an unofficial API for interacting with Gmail's web interface from browser extensions. It let… | 77 | 3848 | active |
| sentient-agi/OpenDeepSearch OpenDeepSearch is an open-source search tool that augments LLMs and reasoning agents with web search capabilities, aiming to match closed-s… | 23 | 3834 | active |
| Findomain/Findomain Findomain is a fast subdomain enumeration tool written in Rust that discovers subdomains via Certificate Transparency logs and APIs without… | 78 | 3793 | active |
| fqscfqj/Y2A-Auto Y2A-Auto is a self-hosted Python application that automates re-uploading YouTube videos to AcFun and bilibili. It handles the full pipeline… | 81 | 3787 | active |
| Mouseww/anything-analyzer An Electron-based all-in-one protocol analysis toolkit that captures HTTP(S) traffic from browsers, desktop apps, terminals, scripts, and m… | 77 | 3734 | active |
| Orion Store Orion Store is a privacy-focused Android app store client that distributes modded apps (like YouTube Morphe) and open-source tools using Gi… | 85 | 3719 | active |
| AmintaCCCP/GithubStarsManager An AI-powered desktop and web application for managing GitHub stars, forks, and Gists with semantic vector search, automatic categorization… | 86 | 3667 | active |
| AIDotNet/OpenDeepWiki OpenDeepWiki is an AI-driven platform that turns Git repositories, ZIP archives, and local directories into searchable, structured document… | 85 | 3615 | active |
| deta/surf Deta Surf is an open-source, local-first AI notebook for macOS, Windows, and Linux that lets users organize files, PDFs, websites, and YouT… | 72 | 3600 | active |
| zhaoyingjun/chatbot A Chinese chatbot project that trains conversational models (Seq2Seq and GPT-style, including small SLM variants) on custom corpora with lo… | 49 | 3582 | active |
| SkyworkAI/DeepResearchAgent DeepResearchAgent is a Python framework for hierarchical multi-agent systems that combines a top-level planning agent with specialized lowe… | 63 | 3552 | active |
| any4ai/AnyCrawl AnyCrawl is a Node.js/TypeScript web crawler and scraping service that converts websites into LLM-ready markdown/JSON data and extracts str… | 85 | 3477 | active |
| ScrapydWeb ScrapydWeb is a self-hosted web application for managing Scrapyd clusters, supporting all Scrapyd JSON APIs, multinode operations, and Scra… | 30 | 3411 | stable |
| Intelligent-Internet/ii-agent II-Agent is an open-source (Apache-2.0) AI agent framework for building and deploying intelligent assistants that execute complex multi-ste… | 62 | 3391 | active |
| Sophomoresty/gemini-web2api A single-file Python server that exposes Google Gemini's web interface as an OpenAI-compatible API, supporting chat completions, streaming,… | 54 | 3346 | active |
| minitap-ai/mobile-use An open-source Python AI agent framework that controls real Android and iOS devices using natural language commands, built on LangGraph. It… | 71 | 3298 | active |
| PastKing/tgbot-verify A self-hosted Python Telegram bot that automates SheerID student/teacher identity verification flows for services like Spotify, ChatGPT, Ge… | 45 | 3293 | active |
| rtcatc/Packer-Fuzzer Packer Fuzzer is a Python-based security scanner that targets websites built with JavaScript module bundlers like Webpack. It automatically… | 23 | 3250 | active |
| hydrusnetwork/hydrus Hydrus Network is a desktop media management application that organizes large file collections with tags instead of folders, in a booru-sty… | 95 | 3238 | active |
| emeryberger/CSrankings CSRankings is a web application that ranks computer science departments worldwide based on faculty publications at selective conferences, u… | 77 | 3216 | active |
| liyown/ai-trend-publish TrendPublish is a TypeScript-based automated content pipeline for WeChat Official Accounts that scrapes multiple sources (Twitter/X, RSS, s… | 76 | 3210 | active |
| crawlab-team/artipub ArtiPub is a self-hosted article publishing platform that automatically distributes content to multiple media channels such as CSDN and Seg… | 48 | 3205 | active |
| power721/alist-tvbox A Java-based proxy server that bridges AList storage with TvBox media players, generating subscription playlists and supporting content sea… | 95 | 3150 | active |
| mitsuhiko/agent-stuff A personal npm package (mitsupi) of skills, extensions, prompt commands, and themes for the Pi Coding Agent, published by Armin Ronacher. I… | 59 | 3148 | active |
| autobrr/autobrr autobrr is a self-hosted download automation tool for torrents and Usenet, written in Go with a built-in web UI. It monitors IRC announce c… | 98 | 3136 | active |
| bestofjs/bestofjs Best of JS is a web application that tracks GitHub star counts daily for a curated list of ~2000 open-source projects related to the web pl… | 67 | 3136 | active |
| LANraragi LANraragi is an open-source web application for archiving, organizing, and reading manga/doujinshi and comics, built on Mojolicious and Red… | 89 | 3124 | active |
| MiroFlow MiroFlow is a performance-first open-source agent framework for building, orchestrating, and benchmarking LLM-powered agents, with MiroThin… | 56 | 3120 | active |
| msgbyte/tianji Tianji is a self-hosted all-in-one insight hub combining website analytics, uptime monitoring, and server status reporting in a single ligh… | 88 | 3099 | active |
| darkoperator/dnsrecon DNSRecon is a Python-based DNS enumeration tool for security assessments and network troubleshooting. It supports zone transfer checks, gen… | 89 | 3080 | active |
| open-rpa/openrpa OpenRPA is a free, open-source, enterprise-grade Robotic Process Automation (RPA) tool with a visual workflow designer for Windows. It can … | 65 | 3074 | active |
| microsoft/playwright-dotnet The official .NET port of Playwright, a library for automating Chromium, Firefox, and WebKit browsers through a single API. It supports end… | 96 | 3023 | active |
| Manisso/fsociety Fsociety is a Python-based penetration testing framework that bundles a menu of hacking tools covering information gathering, password atta… | 70 | 12354 | maintenance |
| zcaceres/markdownify-mcp A Model Context Protocol (MCP) server that converts PDFs, images, audio, Office documents, and web content (including YouTube transcripts a… | 79 | 2998 | active |
| bergside/design-md-chrome A Chrome extension that extracts visual styles (typography, colors, spacing, shadows, motion) from any website and generates DESIGN.md or S… | 62 | 2972 | active |
| palemoky/chinese-poetry-api A high-performance Go-based API service serving nearly 400,000 classical Chinese poems (Tang, Song, Yuan, etc.) with both REST and GraphQL … | 74 | 2930 | active |
| magicblack/maccms10 Apple CMS (MacCMS) v10 is an open-source PHP+MySQL content management system built on the ThinkPHP 5 framework, designed for quickly buildi… | 92 | 2908 | active |
| Suxiaoqinx/Netease_url A self-hosted Python service that parses Netease Cloud Music (网易云音乐) songs, playlists, and albums to retrieve direct download URLs at multi… | 74 | 2864 | active |
| sahibzada-allahyar/YC-Killer A collection of open-source, enterprise-grade AI agents intended as free alternatives to commercial Y Combinator startups. It includes a de… | 60 | 2828 | active |
| mshumer/OpenDeepResearcher A Jupyter Notebook (with a Gradio UI variant) implementing an autonomous AI research agent that iteratively generates search queries, fetch… | 26 | 2795 | active |
| levywang/avhub AvHub is a self-hostable web application for searching and managing R18 (adult) video resources, built with a Vue 3 frontend and a FastAPI … | 53 | 2742 | active |
| heshengtao/super-agent-party Super Agent Party is a self-hosted, all-in-one AI desktop companion that combines VRM-based virtual characters, agent skills, MCP tool supp… | 85 | 2736 | active |
| ShinMegamiBoson/OpenPlanter OpenPlanter is a recursive-language-model investigation agent that ingests heterogeneous datasets (corporate registries, campaign finance, … | 43 | 2680 | active |
| prateek-chaubey/YTPro YTPro is an alternative YouTube client for Android built with a WebView and JavaScript injection, offering ad blocking, background playback… | 80 | 2652 | active |
| firecrawl/open-agent-builder Open Agent Builder is a visual, no-code workflow builder for creating AI agent pipelines powered by Firecrawl web scraping. It provides a d… | 35 | 2639 | active |
| CoderLuii/HolyClaude HolyClaude is a Docker-based AI coding workstation that bundles Claude Code, a web UI, eight AI coding CLIs (Gemini, OpenAI Codex, Cursor A… | 81 | 2579 | active |
| aingdesk/AingDesk AingDesk is an open-source, MIT-licensed AI assistant application for running local AI models (e.g., via Ollama) and third-party model APIs… | 57 | 2539 | active |
| Barre/privaxy Privaxy is a MITM HTTP(S) proxy written in Rust that blocks ads and trackers at the network level, supporting Adblock Plus and uBlock Origi… | 23 | 2537 | active |
| CJackHwang/AIstudioProxyAPI A self-hosted proxy server that exposes Google AI Studio's web chat interface as an OpenAI-compatible API using FastAPI and Camoufox/Playwr… | 64 | 2513 | active |
| Epic Awesome Gamer A Python library that solves hCaptcha challenges using multimodal large language models and ONNX vision models (YOLO, CLIP, ResNet) integra… | 80 | 2508 | active |
| 24mlight/A_Share_investment_Agent A Python-based multi-agent AI investment analysis system for China's A-share market that uses LLMs (Gemini/OpenAI-compatible) with bull/bea… | 56 | 2482 | active |
| SickChill/sickchill SickChill is a self-hosted, cross-platform automatic video library manager (PVR) for TV shows written in Python with a web interface. It wa… | 85 | 2444 | active |
| 6551Team/opennews-mcp An MCP (Model Context Protocol) server that aggregates real-time news and market data from 85+ sources across news, exchange listings, on-c… | 58 | 2425 | active |
| evan-buss/openbooks OpenBooks is a self-hosted application for searching and downloading eBooks from the IRC Highway network, available as a web app (Go server… | 27 | 2421 | active |
| kevinluosl/deepbot DeepBot is a system-level AI assistant desktop application (Electron/TypeScript) that automates enterprise and personal workflows through m… | 52 | 2418 | active |
| learnhouse/learnhouse LearnHouse is a next-generation open-source learning management system (LMS) for creating, sharing, and selling educational content. It com… | 100 | 2404 | active |
| AabyssZG/SpringBoot-Scan SpringBoot-Scan is an open-source penetration testing framework targeting Spring Boot applications, written in Python. It scans for sensiti… | 47 | 2352 | active |
| jxhczhl/JsRpc JsRpc is a Go-based RPC server that lets you remotely invoke JavaScript methods running in a browser via WebSocket, avoiding the need to ex… | 78 | 2350 | active |
| stephengpope/no-code-architects-toolkit A self-hostable Flask-based API that consolidates common media processing tasks—video editing, captioning, audio conversion, transcription,… | 47 | 2345 | active |
| Decohererk/DecoTV DecoTV is a self-hosted, cross-platform video aggregation and playback web application built on Next.js, Tailwind CSS, and TypeScript. It a… | 72 | 2343 | active |
| Laverna/laverna Laverna is a JavaScript note-taking web application with a Markdown editor and client-side encryption, positioned as an open-source alterna… | 23 | 9168 | maintenance |
| jofpin/trape Trape is an OSINT analysis and research tool for tracking people online and executing real-time social engineering attacks, built in Python… | 32 | 9055 | maintenance |
| guy-hartstein/company-research-agent A multi-agent company research application built with LangGraph and Tavily that generates comprehensive due-diligence reports on any compan… | 83 | 2301 | active |
| rushindrasinha/youtube-shorts-pipeline Verticals v3 (repo youtube-shorts-pipeline) is a Python CLI that automates producing and publishing YouTube Shorts: it researches a topic, … | 55 | 2301 | active |
| DanOps-1/Gpt-Agreement-Payment A Python toolkit that reverse-engineers and replays the end-to-end ChatGPT Plus/Team/Pro subscription payment flow (Stripe Checkout, PayPal… | 50 | 2288 | active |
| Weizhena/Deep-Research-skills A structured deep research skill (prompt/workflow package) for Claude Code, OpenCode, and Codex that runs a two-phase research process: out… | 57 | 2287 | active |
| anandprtp/Antra Antra is a desktop music library builder that resolves links from streaming services like Spotify, YouTube Music, Apple Music, Amazon Music… | 78 | 2279 | active |
| axiaoxin-com/investool InvesTool is a Go-based personal investment analysis tool that automates fundamental stock screening, financial report analysis, and mutual… | 30 | 2279 | active |
| spyboy-productions/CloakQuest3r CloakQuest3r is a Python-based open-source security research tool that identifies potential origin IP exposure of websites protected by Clo… | 51 | 2269 | active |
| KikoPlayProject/KikoPlay KikoPlay is a full-featured danmaku (bullet comment) video player built on Qt and libmpv, with danmu fetching from major video sites, a tre… | 88 | 2248 | active |
| tsingyuai/scientify Scientify is an OpenClaw plugin that automates end-to-end scientific research workflows, continuously ingesting new papers, evolving hypoth… | 71 | 2244 | active |
| tramhao/termusic Termusic is a terminal-based (TUI) music and podcast player written in Rust with multiple playback backends (Rusty, MPV, GStreamer). It can… | 82 | 2224 | active |
| rcortx/kiwiq KiwiQ is a production-grade multi-agent orchestration platform in Python that lets you define AI workflows as JSON/Python graph schemas com… | 46 | 2211 | active |
| proma-ai/Proma Proma is a local-first, open-source AI desktop application that combines multi-model chat, a general-purpose agent runtime, workspaces, ski… | 76 | 2208 | active |
| TikTok Live Connector An unofficial Node.js library that connects to TikTok LIVE streams and delivers real-time events such as chat comments, gifts, likes, follo… | 85 | 2188 | active |
| atomicstrata/llm-wiki-compiler llmwiki is a Node.js CLI knowledge compiler that turns raw sources (papers, docs, notes, PDFs, URLs) into an interlinked, citation-traceabl… | 81 | 2169 | active |
| TangSY/echarts-map-demo A Vue-based web application that fetches real-time GeoJSON administrative boundary data for China (province, city, county, town levels) for… | 74 | 2151 | active |
| btahir/open-deep-research An open-source web application that replicates Gemini's Deep Research by searching the web, extracting page content, and generating AI-writ… | 10 | 2142 | active |
| satellitecomponent/Neurite Neurite is an open-source fractal-based mind-mapping workspace that combines the Mandelbrot set interface with graph-of-thought reasoning f… | 57 | 2140 | active |
| kkbo8005/mitan Mitan (密探) is an all-in-one penetration testing and security assessment desktop application integrating asset mapping, subdomain brute-forc… | 92 | 2111 | active |
| iniwap/AIWriteX AIWriteX is a Python-based desktop application that automates WeChat Official Account content creation using CrewAI multi-agent workflows, … | 87 | 2106 | active |
| BetaStreetOmnis/xhs_ai_publisher A desktop application for AI-powered content creation and automated publishing on Xiaohongshu (Rednote), built with PyQt5, FastAPI, and Pla… | 64 | 2086 | active |
| alibaba/anyproxy AnyProxy is a fully configurable HTTP/HTTPS proxy server built on Node.js, supporting custom rule modules to intercept, modify, and inspect… | 32 | 7897 | maintenance |
| Jamailar/Beav Beav (formerly RedBox) is a local-first AI content operations workbench for social media creators, combining a desktop app and a Chrome/Edg… | 82 | 1998 | active |
| wuchangming/spy-debugger spy-debugger is a Node.js CLI tool that provides one-stop remote debugging and packet capture for mobile web pages. It injects weinre debug… | 37 | 7617 | maintenance |
| Atomic Atomic is a self-hosted, local-first personal knowledge base that turns markdown notes into a semantically-connected, AI-augmented knowledg… | 75 | 1988 | active |
| filiksyos/gitreverse GitReverse is a web application that turns any public GitHub repository into a single synthetic prompt that could have been used to vibe-co… | 60 | 1984 | active |
| songloft-org/songloft Songloft (formerly MiMusic) is a free, ad-free, plugin-based self-hosted music server written in Go, designed for managing music you legall… | 84 | 1983 | active |
| baoweise-bot/aimili-vpngate AimiliVPN is a Python tool that manages VPNGate public VPN nodes on Linux VPS, providing node fetching/testing, connection switching, and a… | 76 | 1976 | active |
| helallao/perplexity-ai An unofficial Python wrapper for Perplexity.ai that reverse-engineers the web interface, supporting sync and async APIs, streaming response… | 74 | 1976 | active |
| firecrawl/fireplexity Fireplexity is an open-source, Perplexity-style AI search engine built with TypeScript that delivers answers with real-time citations, stre… | 33 | 1975 | active |
| oficcejo/aiagents-stock A Python multi-AI-agent stock analysis and monitoring system for mainland China A-shares, simulating a team of securities analysts to produ… | 52 | 1945 | active |
| liuhuanyong/QASystemOnMedicalKG A tutorial-style Python project that builds a disease-centered medical knowledge graph (44k entities, 300k relations) in Neo4j from scraped… | 32 | 7398 | maintenance |
| Dyneteq/reconya reconYa is a self-hosted network reconnaissance and asset discovery tool written in Go that scans local networks via ICMP, TCP, and ARP to … | 81 | 1902 | active |
| ginlix-ai/LangAlpha LangAlpha is an open-source agentic AI harness for financial market research and investment decision support, inspired by code-agent harnes… | 83 | 1880 | active |