Open-source web-scraping software
2000 open-source libraries, tools and applications tagged web-scraping, ranked by how closely web-scraping is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
2000 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| Kspider Kspider is a self-hosted visual web scraping platform written in Java where users define crawler workflows as flowcharts without writing ba… | 15 | 1265 | active |
| ImAiiR/QobuzDownloaderX QobuzDownloaderX (QBDLX) is a desktop GUI application that downloads music streams directly from the Qobuz streaming platform using its API… | 72 | 1264 | active |
| egbertbouman/youtube-comment-downloader A Python script and library for downloading YouTube video comments without using the official YouTube API. It outputs comments in JSONL, JS… | 82 | 1260 | active |
| AIPexStudio/AIPex AIPex is an open-source (MIT) AI browser automation assistant delivered as a Chrome/Edge extension that runs inside your existing browser w… | 76 | 1256 | active |
| MarioVilas/googlesearch A Python library that performs Google web searches programmatically and returns result URLs, without using an official Google API. It is un… | 10 | 1254 | active |
| myreader-io/myGPTReader myGPTReader is a Slack bot powered by ChatGPT that reads and summarizes webpages, documents (eBooks, PDF, DOCX), and YouTube videos, and su… | 10 | 4414 | maintenance |
| tomorrow505/auto_feed_js A Tampermonkey/Greasemonkey userscript that enables one-click cross-posting (re-seeding) of torrents across 100+ private tracker sites. It … | 75 | 1252 | active |
| gooseworks-ai/goose-skills A library of 200+ AI agent skills and data APIs for growth and go-to-market work, installable into coding agents like Claude Code, Cursor, … | 60 | 1252 | active |
| EmergenceAI/Agent-E Agent-E is an open-source agent-based system that automates actions on the user's computer, currently focusing on browser automation via na… | 58 | 1252 | active |
| Decodo Decodo (formerly Smartproxy) is a commercial rotating proxy network and web scraping platform offering 125M+ residential, mobile, ISP, and … | 65 | 1246 | active |
| firecrawl/web-agent An open-source TypeScript framework for building autonomous web research agents, layered from a Next.js chat template down to an agent core… | 47 | 1246 | active |
| jeremykendall/php-domain-parser A PHP library that parses domain names into their component parts (subdomain, registrable domain, second-level domain, public suffix) using… | 51 | 1245 | active |
| WebScrapBook WebScrapBook is a browser extension that captures web pages faithfully into various archive formats (including MAFF and HTZ) for local or b… | 76 | 1243 | active |
| anvaka/npmgraph.an A web application that visualizes the dependency graph of npm packages in 2D and 3D, built with Vue 3 and the ngraph library. It fetches pa… | 63 | 1243 | active |
| raawaa/jav-scrapy A TypeScript-based Node.js CLI tool that batch-scrapes JAV (adult video) metadata, magnet links, and cover images from source websites. It … | 90 | 1242 | active |
| seveniruby/AppCrawler AppCrawler is a Scala-based automated app traversal (crawler) testing tool built on Appium, supporting Android, iOS, mini-programs, and Har… | 54 | 1241 | active |
| CERT-Polska/Artemis Artemis is a modular, open-source vulnerability scanner developed by CERT Polska that checks website security at scale. It automatically ge… | 94 | 1240 | active |
| warifp/FacebookToolkit A PHP command-line toolkit for retrieving Facebook account data via the Graph API, including access tokens, friend IDs, emails, and names, … | 23 | 1235 | active |
| saeeddhqan/Maryam OWASP Maryam is a modular open-source OSINT framework for harvesting data from open sources, search engines, and social networks. It provid… | 10 | 1231 | active |
| taranis-ai/taranis-ai Taranis AI is a self-hosted open-source OSINT platform that collects news articles from web sources and uses NLP/AI to enrich, cluster, and… | 93 | 1228 | active |
| cnbattle/douyin A Go-based crawler that scrapes Douyin (TikTok China) recommendation and search page video lists by controlling the mobile app on a real de… | 57 | 1225 | active |
| GPTaku Plugins insane-search is a Claude Code plugin that reads public web pages that would otherwise be blocked (403, CAPTCHA, WAF), escalating through p… | 83 | 1224 | active |
| mika-cn/maoxian-web-clipper MaoXian Web Clipper is a browser extension for Firefox and Chrome that clips selected content from web pages and saves it to the local mach… | 64 | 1224 | active |
| kmille/deezer-downloader A self-hosted Python service with a simple web frontend for downloading songs, albums, and playlists from Deezer (and via yt-dlp), with ID3… | 77 | 1221 | active |
| hengliyin/cdfang-spider A full-stack web application that scrapes Chengdu Housing Association lottery housing data and presents it through interactive charts and s… | 49 | 1221 | active |
| eooce/Auto-login-netlib A JavaScript automation script that periodically logs into netlib.re accounts to keep free domains alive, run via GitHub Actions on a 60-da… | 36 | 1220 | active |
| ycdxsb/PocOrExp_in_Github A Python CLI tool that automatically aggregates proof-of-concept (POC) and exploit (EXP) code from GitHub by CVE ID, using CVE information … | 77 | 1218 | active |
| Sak32009/GetDataFromSteam-SteamDB A userscript that extracts game and DLC data from Steam store pages and SteamDB. It runs via userscript managers like Tampermonkey or Viole… | 74 | 1214 | active |
| browserable/browserable Browserable is an open-source, self-hostable JavaScript library for building AI browser agents that navigate sites, fill out forms, click b… | 34 | 1210 | active |
| N0rz3/Phunter Phunter is a Python CLI OSINT tool that gathers information about phone numbers, including operator, line type, location, reputation, spam … | 27 | 1203 | active |
| teaSummer/MCiSEE MCiSEE is a web application that aggregates and presents Minecraft resources (software, tools, websites) collected from the internet in an … | 66 | 1201 | active |
| gityuanbao/share A personal open-source repository whose main component is akshare_collector, a Python tool built on AKShare that collects Chinese financial… | 61 | 1200 | active |
| MediaBox MediaBox is an Android 'universal media container' app that aggregates video, manga, and other media through a WeChat-mini-program-like plu… | 23 | 1200 | active |
| internetwache/GitTools GitTools is a collection of three shell/Python scripts for finding and exploiting websites that publicly expose their .git directory. It in… | 66 | 4191 | maintenance |
| scrapinghub/splash Splash is a lightweight, scriptable headless browser exposed as a service with an HTTP API, implemented in Python 3 using Twisted and Qt5. … | 32 | 4190 | maintenance |
| neon-mmd/websurfx Websurfx is an open-source meta search engine written in Rust that aggregates results from multiple search engines while respecting user pr… | 82 | 1199 | active |
| xenova/chat-downloader Chat Downloader is a Python tool and library for retrieving chat messages from livestreams, videos, clips, and past broadcasts on platforms… | 42 | 1198 | active |
| niudaii/zpscan zpscan is a Go-based command-line information gathering and reconnaissance tool for security assessments. It bundles subdomain enumeration,… | 23 | 1197 | active |
| instantbox/instantbox instantbox is a self-hosted service that spins up temporary, clean Linux containers (Ubuntu, CentOS, Arch, Debian, Fedora, Alpine) with ins… | 32 | 4172 | maintenance |
| constverum/ProxyBroker ProxyBroker is an asynchronous Python tool that finds public HTTP(S) and SOCKS4/5 proxies from ~50 sources and concurrently checks their ty… | 32 | 4169 | maintenance |
| austin-weeks/miasma Miasma is a lightweight Rust web server that traps AI web scrapers in an endless pit of poisoned training data and self-referential links. … | 80 | 1194 | active |
| aooiuu/any-reader Any-Reader is an open-source, cross-platform content aggregation tool that unifies novels, manga, video, and audio from user-defined source… | 72 | 1191 | active |
| jayus0821/swagger-hack A Python CLI tool that automatically crawls all endpoints exposed by leaked Swagger/OpenAPI documentation and sends configured test request… | 53 | 1189 | active |
| intoli/user-agents A JavaScript/TypeScript npm package for generating random user agents weighted by real-world market share, with daily-updated data. It also… | 77 | 1186 | active |
| auto-novel/auto-novel AutoNovel is a website application that automatically machine-translates light novels (web novels, bunko novels, and local files) using LLM… | 76 | 1185 | active |
| scdl-org/scdl scdl is a Python command-line tool for downloading music, playlists, likes, and reposts from SoundCloud, with automatic ID3 metadata taggin… | 63 | 4126 | maintenance |
| shobrook/rebound Rebound is a command-line tool that runs your file and, when an exception is thrown, instantly fetches related Stack Overflow questions and… | 23 | 4115 | maintenance |
| Evolution0/bandcamp-dl A Python command-line tool for downloading albums and tracks from bandcamp.com, with options for filename templating, album art embedding, … | 67 | 1179 | active |
| grangier/python-goose Python-Goose is a Python library that extracts the main body text, metadata, top image, and embedded videos from news article web pages. It… | 61 | 4106 | maintenance |
| rchipka/node-osmosis Osmosis is an HTML/XML parser and web scraper library for Node.js built on native libxml C bindings. It offers a chainable, promise-like in… | 32 | 4104 | maintenance |
| rverton/webanalyze webanalyze is a Go port of Wappalyzer that detects the technologies used on websites, built for performant mass scanning of large host list… | 96 | 1178 | active |
| datawhores/OF-Scraper OF-Scraper is a command-line tool for downloading media from OnlyFans and performing bulk actions like liking or unliking posts. It is a re… | 75 | 1178 | active |
| Abbey ScrapeServ is a self-hosted API service that accepts a URL and returns the website's data along with browser screenshots, using Playwright … | 22 | 1176 | active |
| ycngmn/Nobook Nobook is a lightweight, ad-free Android client for browsing Facebook, built with Kotlin and Jetpack Compose on top of a WebView. It blocks… | 10 | 1176 | active |
| apache/groovy-geb Apache Geb is a Groovy-based browser automation library built on WebDriver, combining jQuery-like content selection with Page Object modell… | 76 | 1174 | active |
| online-judge-tools A command-line tool that automates solving problems on online judges like AtCoder, Codeforces, and HackerRank. It downloads sample and syst… | 23 | 1173 | active |
| evyatarmeged/Raccoon Raccoon is a Python-based offensive security CLI tool for reconnaissance and information gathering. It performs DNS lookups, WHOIS, TLS ana… | 64 | 4047 | maintenance |
| asdfghj1237890/WebVideo2NAS A self-hosted pipeline consisting of a Chrome extension and a Dockerized FastAPI backend that detects HLS (M3U8), DASH (MPD), MP4, and MOV … | 82 | 1164 | active |
| xeco23/WasIstLos WasIstLos is an unofficial WhatsApp desktop client for Linux, written in C++ using gtkmm and WebKitGTK to wrap WhatsApp Web. It adds deskto… | 10 | 1164 | active |
| zu1k/proxypool A Go service that automatically crawls proxy nodes (ss, ssr, vmess, trojan) from Telegram channels, subscription URLs, and the public inter… | 23 | 4034 | maintenance |
| EmilStenstrom/justhtml JustHTML is a pure Python HTML5 parser with browser-style error recovery, safe-by-default sanitization, CSS selector querying, and serializ… | 84 | 1159 | active |
| Libraries.io Libraries.io is an open source discovery service that indexes millions of packages across 32 package managers, letting developers search an… | 76 | 1159 | active |
| CMHopeSunshine/LittlePaimon LittlePaimon is a multifunctional Genshin Impact chatbot built on the NoneBot2 framework, supporting the OneBot protocol for QQ. It provide… | 24 | 1159 | active |
| 0x727/ShuiZe_0x727 ShuiZe_0x727 is a Python-based automated information gathering (reconnaissance) tool for red team operators. Given a root domain, C-segment… | 23 | 4021 | maintenance |
| yunginnanet/HellPot HellPot is a cross-platform HTTP honeypot that punishes bots ignoring robots.txt by streaming an infinite Markov-chain-generated page of ps… | 45 | 1157 | active |
| random-robbie/My-Shodan-Scripts A collection of Python 3 scripts for querying the Shodan search engine to find exposed devices and services on the internet. It bundles man… | 57 | 1156 | active |
| fanpei91/torsniff torsniff is a Go CLI tool that sniffs torrent metadata from the BitTorrent network by participating in the DHT and connecting to peers to d… | 10 | 4009 | maintenance |
| webrecorder/browsertrix-crawler Browsertrix Crawler is a standalone browser-based high-fidelity web crawling system that runs in a single Docker container. It uses Puppete… | 98 | 1151 | active |
| 0xsha/CloudBrute CloudBrute is a Go CLI tool that enumerates a company's infrastructure, files, and applications across major cloud providers (Amazon, Googl… | 24 | 1151 | active |
| filipedeschamps/rss-feed-emitter A Node.js library that aggregates RSS and Atom news feeds and emits events for every new item published. It automatically manages feed hist… | 68 | 1150 | stable |
| AndyTheFactory/newspaper4k Newspaper4k is a Python library and CLI for scraping and curating news articles, extracting text, titles, authors, publish dates, and metad… | 78 | 1149 | active |
| nicolomantini/LinkedIn-Easy-Apply-Bot A Python bot that automates applying to jobs on LinkedIn using the Easy Apply feature. It reads search preferences, resume uploads, and bla… | 53 | 1149 | active |
| mrkrsl/web-search-mcp A locally hosted MCP (Model Context Protocol) server written in TypeScript that gives local LLMs web search capabilities without requiring … | 33 | 1147 | active |
| tmwgsicp/wechat-download-api An open-source API service for fetching WeChat official account articles, generating standard RSS 2.0 feeds, and exporting entire account a… | 72 | 1146 | active |
| agentbay-ai/wuying-agentbay-sdk Multi-language SDK (Python, TypeScript, Go, Java) for Wuying AgentBay, Alibaba Cloud's cloud sandbox platform built for AI agents. It lets … | 70 | 1146 | active |
| Aggr AGGR (SignificantTrade) is a Vue.js web application that aggregates live cryptocurrency trades from many exchanges (Binance, Coinbase, BitM… | 56 | 1146 | active |
| eshaham/israeli-bank-scrapers A TypeScript library providing scrapers for all major Israeli banks and credit card companies, published as the npm package israeli-bank-sc… | 98 | 1145 | active |
| platonai/Browser4 Browser4 is an AI-native browser engine built in Kotlin for autonomous agents, intelligent data extraction, and large-scale web automation.… | 93 | 1145 | active |
| hasanfirnas/symbiote Symbiote is a Python-based social engineering tool that generates a phishing page to trick a target into granting camera permission, then c… | 33 | 1145 | active |
| Haleydu/Cimoc Cimoc is an open-source Android manga/comic reader app that aggregates multiple online comic sources. It supports page and scroll reading m… | 23 | 3966 | maintenance |
| kevthehermit/PasteHunter PasteHunter is a Python 3 application that queries public pastebin-style sites (pastebin.com, GitHub gists, slexy, stackexchange, etc.) and… | 47 | 1143 | active |
| Fess Fess is an open-source, self-hosted enterprise search server built on OpenSearch with a browser-based admin UI, built-in crawlers for web s… | 95 | 1141 | active |
| Junyi-99/ChatGPT-API-Scanner A Python CLI tool that scans GitHub for publicly leaked OpenAI API keys using Selenium browser automation. It is intended for security rese… | 48 | 1137 | active |
| fwonggh/Bthub Bthub is a magnet link and torrent search engine, and this repository serves as its official address release page listing current and backu… | 76 | 1136 | active |
| rebane2001/xikipedia Xikipedia is a web application that presents Wikipedia content as a social media feed, using a simple non-ML algorithm that learns user eng… | 50 | 1136 | active |
| ChineseSubFinder/ChineseSubFinder A self-hosted Go application that automatically downloads Chinese subtitles for movies and TV shows from subtitle websites. It integrates w… | 23 | 3929 | maintenance |
| GoAnime GoAnime is a terminal-based (TUI) anime browser written in Go that lets users search, stream, and download anime episodes directly in mpv. … | 91 | 1135 | active |
| Epic Awesome Gamer A Python application that automatically claims weekly free games and monthly content from the Epic Games Store. It uses browser automation … | 53 | 1129 | active |
| johnwmillr/LyricsGenius A Python client library that wraps the Genius.com API and scrapes song lyrics, artist, and album metadata. It provides simple search and do… | 95 | 1127 | active |
| bellingcat/auto-archiver A Python tool by Bellingcat that automatically archives web content such as videos, images, social media posts, and webpages from URLs supp… | 95 | 1126 | active |
| elixir-crawly/crawly Crawly is a high-level web crawling and scraping framework for Elixir, modeled after Scrapy, where developers define spiders that fetch pag… | 34 | 1122 | active |
| shanmiteko/LotteryAutoScript A Node.js automation script that monitors Bilibili dynamic posts for lottery/giveaway draws and automatically participates by liking, comme… | 83 | 1121 | active |
| projectdiscovery/wappalyzergo A high-performance Go library that ports the Wappalyzer technology detection stack, identifying web technologies from HTTP headers and HTML… | 96 | 1120 | active |
| iFurySt/RedNote-MCP A Model Context Protocol (MCP) server that lets AI clients access RedNote (Xiaohongshu) content, including keyword search and note retrieva… | 16 | 1120 | active |
| Komet/MediaElch MediaElch is a cross-platform desktop media manager for Kodi that scrapes and organizes metadata for movies, TV shows, concerts, and music.… | 64 | 1119 | active |
| r3nt0n/bopscrk bopscrk is a Python CLI tool that generates smart, targeted wordlists for password cracking, combining user-provided words with transformat… | 23 | 1119 | active |
| JimmXinu/FanFicFare FanFicFare is a Python tool that downloads stories from over 100 fanfiction and web fiction sites and converts them into EPUB (and HTML) eB… | 99 | 1116 | active |
| Doriandarko/make-it-heavy A Python framework that emulates Grok Heavy-style deep analysis by orchestrating multiple specialized AI agents in parallel via OpenRouter'… | 30 | 1116 | active |
| leovan/SciHubEVA SciHubEVA is a cross-platform GUI application for searching and downloading papers from Sci-Hub, built with Python and Qt (PySide6/QML). It… | 67 | 1114 | active |
| gautamkrishnar/socli SoCLI is a Python command-line client for Stack Overflow that lets developers search and browse questions and answers without leaving the t… | 52 | 1112 | active |