Open-source web-scraping software
2000 open-source libraries, tools and applications tagged web-scraping, ranked by how closely web-scraping is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
2000 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| w12scan w12scan is a self-hosted network asset discovery engine that aggregates scan results into a searchable web interface backed by Elasticsearc… | 23 | 1330 | maintenance |
| bugswriter/tuxi Tuxi is a shell-based CLI assistant that scrapes Google search results and SERPs to return instant, concise answers to natural language que… | 10 | 1328 | maintenance |
| csns1/bypass-paywalls-chrome-clean-magnolia1234 A Chromium browser extension that bypasses paywalls on supported news sites, allowing users to read articles for free. It supports custom s… | 32 | 1321 | maintenance |
| heremaps/harp.gl harp.gl is an open-source 3D map rendering engine written in TypeScript and built on three.js/WebGL. It enables themeable, high-performance… | 10 | 1316 | maintenance |
| devanshbatham/FavFreak A Python CLI tool that fetches favicon.ico files from lists of URLs, computes their mmh3 hashes, and groups domains/subdomains/IPs by match… | 23 | 1314 | maintenance |
| c4tcom/Katana Katana-ds is a Python CLI tool that automates advanced Google queries known as Google Dorks (Google Dorking), with optional Tor support for… | 10 | 1310 | maintenance |
| InstaPy/instagram-profilecrawl A Python script from the InstaPy ecosystem that crawls public Instagram profile information such as post counts, follower counts, and post … | 32 | 1308 | maintenance |
| danielcardeenas/sulla Sulla is a JavaScript/TypeScript library providing a high-level API to control WhatsApp via Puppeteer, enabling automated responses and mes… | 23 | 1307 | maintenance |
| Sniper970119/dianping_spider A Python web scraper for Dianping (大众点评) that extracts search results, shop details, and reviews, handling dynamic font encryption without … | 69 | 1295 | maintenance |
| honzaap/GithubCity A web application that generates a 3D city from your GitHub contribution graph using Three.js. Buildings represent contributions, letting u… | 32 | 1295 | maintenance |
| LangziFun/LangSrcCurise LangSrcCurise is a Django-based automated subdomain asset monitoring system for security researchers tracking SRC (Security Response Center… | 32 | 1292 | maintenance |
| ytdl node-ytdl-core is a JavaScript library for downloading YouTube videos in Node.js, exposing a stream-friendly API with format selection, byt… | 41 | 1290 | maintenance |
| manu354/cryptocurrency-arbitrage A Node.js cryptocurrency arbitrage opportunity calculator that scans over 800 currencies across 50 markets, comparing prices to find the hi… | 23 | 1284 | maintenance |
| sunra/php-simple-html-dom-parser A PHP library adapting Simple HTML DOM Parser for Composer and PSR-0, allowing easy HTML parsing and manipulation. It supports invalid HTML… | 23 | 1283 | maintenance |
| marcelja/facebook-delete A Go CLI tool that bulk-deletes Facebook activity (posts, comments, likes, etc.) without deleting the whole account, using rate-limited req… | 10 | 1281 | maintenance |
| useragents/Zefoy-TikTok-Automator A Python/Selenium automation script that drives zefoy.com to send TikTok followers, views, likes, shares, favorites, and comment likes. It … | 32 | 1278 | maintenance |
| Song-Li/cross_browser A research project from Johns Hopkins implementing cross-browser fingerprinting that identifies users across different browsers on the same… | 32 | 1268 | maintenance |
| DaidoujiChen/Dai-Hentai Dai-Hentai is an iOS app for browsing, reading, and favoriting content from the e-Hentai/EXHentai adult manga sites. It is written in Objec… | 23 | 1267 | maintenance |
| goldfishh/chatgpt-tool-hub An open-source Python tool engine that lets ChatGPT (and other LLMs) invoke tools like web search, math computation, code execution, and co… | 21 | 1264 | maintenance |
| Beanbun Beanbun is a multi-process web crawler framework written in PHP, built on Workerman with Guzzle as the default downloader. It supports dist… | 57 | 1261 | maintenance |
| dchrastil/ScrapedIn A Python CLI tool that scrapes LinkedIn without API restrictions to enumerate employees of a target company for red team or social engineer… | 32 | 1241 | maintenance |
| CyberPunkMetalHead/gateio-crypto-trading-bot-binance-announcements-new-coins A Python crypto trading bot that scrapes the Binance Announcements page for new coin listings and automatically places buy orders on Gate.i… | 32 | 1236 | maintenance |
| istresearch/scrapy-cluster Scrapy Cluster is a distributed web scraping framework built on Scrapy that uses Redis to coordinate crawl requests and Kafka as a data bus… | 10 | 1222 | maintenance |
| vysecurity/LinkedInt LinkedInt is a Python CLI tool for LinkedIn reconnaissance that scrapes employee profiles for a target company and generates an HTML report… | 10 | 1214 | maintenance |
| yqchilde/JDMemberCloseAccount A Python/Selenium-based automation tool that automatically quits all JD (Jingdong) shop memberships. It handles SMS verification codes via … | 23 | 1208 | maintenance |
| Viralmaniar/I-See-You ISeeYou is a Bash and JavaScript tool that captures a target's exact GPS coordinates (latitude/longitude) during social engineering or phis… | 32 | 1206 | maintenance |
| KnockOutEZ/wigolo wigolo is a local-first MCP server that gives AI coding agents web intelligence tools: search across 18 engines, fetch with anti-bot escala… | 78 | 5486 | experimental |
| Mapaler/PixivUserBatchDownload A userscript (browser extension-style script) that batch-downloads all public works of a Pixiv artist directly from the Pixiv website, send… | 44 | 1198 | maintenance |
| yhy0/github-cve-monitor A Python application that monitors GitHub every few minutes for newly published CVEs, security tool updates, and activity on watched reposi… | 32 | 1196 | maintenance |
| bonfy/github-trending A Python script that scrapes GitHub's trending page daily and archives the most popular repositories as Markdown files. It is designed to r… | 76 | 1189 | maintenance |
| mkoehnke/WKZombie WKZombie is a Swift headless browser framework for iOS and macOS that navigates websites and collects data without a user interface or API,… | 23 | 1186 | maintenance |
| timwhitez/crawlergo_x_XRAY A Python glue script that combines the crawlergo dynamic crawler with the XRAY passive vulnerability scanner, replaying crawled URLs throug… | 32 | 1182 | maintenance |
| yutto-dev/bilili bilili is a Python CLI tool for downloading Bilibili videos (including bangumi/season content) along with danmaku comments and subtitles. I… | 10 | 1175 | maintenance |
| MLNLP-World/AI-Paper-Collector A tool from the MLNLP community that automatically collects AI-related papers from sources like DBLP, ACL Anthology, NIPS, and OpenReview, … | 32 | 1173 | maintenance |
| knadh/tg-archive tg-archive is a Python CLI tool that syncs Telegram group chat messages into a local SQLite database and generates a static website archive… | 61 | 1172 | maintenance |
| Lucifer1993/SatanSword SatanSword is a Python-based red team penetration testing framework that integrates web fingerprinting, PoC-based vulnerability detection, … | 32 | 1171 | maintenance |
| yorickpeterse/oga Oga is an XML/HTML parser library for Ruby with a simple API for parsing, modifying, and querying documents via XPath. It requires no syste… | 58 | 1169 | maintenance |
| WebSpiderUtils/verification_code A research repository documenting approaches and code for solving mainstream CAPTCHA systems such as Geetest, NetEase Yidun, and Aliyun CAP… | 32 | 1168 | maintenance |
| tongcheng-security-team/NextScan NextScan (飞刃) is an enterprise-grade distributed black-box vulnerability scanning platform built in Go, composed of Server, Agent, and Web … | 21 | 1164 | maintenance |
| dixudx/tumblr-crawler A Python script that downloads all photos and videos from specified Tumblr blogs. It supports batch site lists, proxy configuration, and sk… | 55 | 1160 | maintenance |
| holgerd77/django-dynamic-scraper A Django app that builds on the Scrapy framework and lets you create and manage Scrapy spiders through the Django admin interface. It remov… | 23 | 1154 | maintenance |
| jlobos/instagram-web-api A Node.js client library for Instagram's private web API, supporting login, profile access, photo upload, likes, comments, follows, and sea… | 23 | 1154 | maintenance |
| s045pd/DarkNet_ChineseTrading A Python-based real-time crawler that monitors Chinese-language darknet marketplaces over Tor, with automatic account registration, login, … | 10 | 1151 | maintenance |
| devxprite/infoooze Infoooze is a Node.js-based OSINT (Open-Source Intelligence) CLI tool for quickly gathering information about websites, IP addresses, usern… | 23 | 1138 | maintenance |
| charlesbel/Microsoft-Rewards-Farmer A Python bot that uses Selenium to automatically earn Microsoft Rewards points by completing searches and daily tasks. It supports multiple… | 10 | 1136 | maintenance |
| yize8888/maotai A Node.js collection of scheduled reservation scripts for the i-Moutai app and several Guizhou e-commerce platforms (Huluxia group-buying s… | 21 | 1135 | maintenance |
| moyada/stealer A self-hosted Python web application that parses and downloads watermark-free videos from Chinese short-video platforms such as Douyin, Kua… | 32 | 1130 | maintenance |
| juancarlospaco/faster-than-requests A Python 3 HTTP client library claiming to be much faster than the popular Requests library, implemented with a compiled core and minimal d… | 67 | 1127 | maintenance |
| kohlschutter/boilerpipe boilerpipe is a Java library for removing boilerplate (ads, navigation, headers) from HTML pages and extracting the main full text content.… | 32 | 1127 | maintenance |
| filamentgroup/glyphhanger Glyphhanger is a Node.js CLI tool for web font optimization. It crawls web pages to determine which unicode-ranges are actually used, and c… | 10 | 1123 | maintenance |
| EricJMarti/inventory-hunter A Python-based stock monitoring bot that watches retail websites for inventory of items like CPUs, GPUs, and game consoles, and sends notif… | 32 | 1122 | maintenance |
| medialab/artoo artoo.js is a JavaScript library injected into a webpage's context (typically via a bookmarklet) that provides client-side web scraping uti… | 23 | 1117 | maintenance |
| KathanP19/JSFScan.sh JSFScan.sh is a shell script that automates JavaScript reconnaissance for bug bounty hunting. Given a list of subdomains, it gathers JS fil… | 32 | 1109 | maintenance |
| cezheng/Fuzi Fuzi is a fast, lightweight Swift library for parsing XML and HTML documents, built on libxml2 as a Swift port of Ono. It provides a modern… | 23 | 1102 | maintenance |
| QuiteRSS QuiteRSS is an open-source cross-platform RSS/Atom news feed reader written in C++ with Qt. It provides a fast, comfortable desktop interfa… | 23 | 1090 | maintenance |
| admintony/Prepare-for-AWD A collection of Python and PHP scripts for AWD (Attack with Defense) CTF competitions, including batch attack scripts for planting and trig… | 32 | 1082 | maintenance |
| exorde-labs/exorde-client The Exorde client is a Python CLI worker node for the Exorde Network, a decentralized protocol where participants scrape social media and w… | 54 | 1081 | maintenance |
| agentcoinorg/evo.ninja Evo.ninja is a versatile generalist AI agent that adapts in real time by selecting pre-defined agent personas (researcher, CSV analyst, dev… | 29 | 1080 | maintenance |
| acikyazilimagi/afet-org An open-source earthquake relief platform (depremyardim.com / afetharita.com) that aggregates calls for help from Twitter, WhatsApp, Telegr… | 31 | 1077 | maintenance |
| wireghoul/htshells A collection of self-contained .htaccess files that turn Apache servers into web shells or launch various attacks when uploaded. It include… | 32 | 1075 | maintenance |
| metachris/pdfx PDFx is a Python command-line tool and library that extracts metadata, text, and references (PDFs, URLs, DOIs, arXiv IDs) from PDF document… | 10 | 1075 | maintenance |
| zaytoun/scihub.py An unofficial Python API and command-line tool for Sci-Hub that can search Google Scholar and download academic papers by DOI, PMID, or URL… | 32 | 1069 | maintenance |
| fuergaosi233/gitbook2pdf A Python CLI tool that asynchronously crawls GitBook documentation sites and converts their contents into a single PDF. It preserves the or… | 10 | 1067 | maintenance |
| jonbakerfish/TweetScraper TweetScraper is a Scrapy-based crawler that scrapes tweets and user information from Twitter Search without using Twitter's official APIs. … | 23 | 1063 | maintenance |
| daprofiler/DaProfiler DaProfiler is a Python-based OSINT CLI tool that traces a person's digital identity from a first and last name, recovering email addresses,… | 10 | 1050 | maintenance |
| zepen/predict_Lottery_ticket A Python application that scrapes historical draw data for Chinese lotteries (双色球/Double Color Ball and 大乐透/Super Lotto) and trains deep le… | 75 | 1043 | maintenance |
| maanavshah/stock-market-india A Node.js/Express API service that fetches and serves stock market data from India's NSE and BSE exchanges. It exposes REST endpoints for m… | 32 | 1042 | maintenance |
| olyble/callPhoneBoom A Python script that automates browser-based form submissions (via Selenium/DrissionPage) against scraped business websites to spam a targe… | 22 | 1042 | maintenance |
| averagesecurityguy/scripts A collection of Python scripts written for use during penetration testing engagements, organized into categories like brute forcing, enumer… | 32 | 1037 | maintenance |
| falling-sky/source Falling Sky is the source code for the test-ipv6.com website, a browser-based IPv6 connectivity testing service built mostly with HTML and … | 67 | 1036 | maintenance |
| yassineaboukir/sublert Sublert is a Python CLI security and reconnaissance tool that uses certificate transparency logs to monitor new subdomains issued TLS/SSL c… | 32 | 1034 | maintenance |
| Malfrats/xeuledoc A Python CLI tool that fetches information about public Google documents across Drive, Docs, Sheets, Slides, and other Google Workspace ser… | 32 | 1032 | maintenance |
| gdtool/zhaopp A Google Drive search engine built on Elasticsearch that indexes millions of files from shared team drives and lets users search and direct… | 32 | 1031 | maintenance |
| supzhang/epg A Python/Django-based system that scrapes TV program schedules (EPG) from multiple Chinese and Hong Kong/Taiwan sources and generates XMLTV… | 31 | 1029 | maintenance |
| admintony/svnExploit SvnExploit is a Python CLI tool that exploits SVN source code disclosure vulnerabilities, supporting both SVN <1.7 and >1.7 repository form… | 32 | 1027 | maintenance |
| ReversecLabs/awspx awspx is a graph-based security tool that visualizes effective access and resource relationships in AWS environments. It resolves IAM polic… | 23 | 1018 | maintenance |
| utkarshkukreti/select.rs A Rust library for extracting useful data from HTML documents, built around predicate-based DOM queries. It is designed for web scraping ta… | 34 | 1017 | maintenance |
| bit4woo/teemo Teemo is a Python command-line reconnaissance tool that collects domains, subdomains, and email addresses for a target organization. It agg… | 32 | 1017 | maintenance |
| s7ckTeam/Glass Glass is a Python CLI tool for rapid fingerprint identification of asset lists, querying Fofa, ZoomEye, Shodan, and 360 Quake APIs to gathe… | 32 | 1012 | maintenance |
| TideSec/FuzzScanner FuzzScanner is a Ruby/Python-based reconnaissance toolset that batch-collects information about target websites, including subdomains, open… | 32 | 1007 | maintenance |
| anvaka/vs A web application that visualizes Google's autocomplete suggestions as an interactive graph. It renders search query expansions as connecte… | 68 | 1003 | maintenance |
| Algebra-FUN/WeReadScan A Python library that uses Selenium headless browsers to scan purchased books from WeRead (WeChat Reading) and convert them into local PDF … | 32 | 1003 | maintenance |
| bitmagnet-io/bitmagnet bitmagnet is a self-hosted BitTorrent indexer, DHT crawler, content classifier and torrent search engine with a web UI, GraphQL API and Ser… | 59 | 4395 | experimental |
| Ekultek/Zeus-Scanner Zeus-Scanner is a Python-based advanced reconnaissance utility for web application security testing. It automates dork-based URL discovery … | 23 | 999 | maintenance |
| gsh199449/spider Gather Platform is a Java-based configurable web crawler built on the WebMagic kernel, with a web console for configuring and managing scra… | 57 | 996 | maintenance |
| browser-use/workflow-use Workflow Use is a Python tool by the Browser Use team for recording and running deterministic browser workflows with variables, falling bac… | 69 | 4211 | experimental |
| MIgHTy-alIeN/ai-trader-bot A Solidity smart contract plus Python automation script that claims to find and execute arbitrage opportunities between Uniswap pools and r… | 58 | 2790 | experimental |
| JSREI/ast-hook-for-js-RE A browser memory roaming tool for JavaScript reverse engineering that hooks variable assignments via AST-transformed proxy responses. It le… | 23 | 1911 | experimental |
| xyzeva/k-id-age-verifier A TypeScript tool that automatically completes K-ID age verification as an adult on platforms like Discord, Twitch, Kick, and Quora. It aut… | 59 | 1769 | experimental |
| richardyc/Chrome-GPT Chrome-GPT is an experimental AutoGPT agent that uses LangChain and Selenium to control a full Chrome browser session on your desktop. It c… | 30 | 1743 | experimental |
| tinyfish-io/bigset-oss BigSet is a self-hostable application that turns a natural-language sentence into a structured, regularly refreshed dataset by dispatching … | 75 | 1706 | experimental |
| PedroHBessa/backscan A Node.js web application that serves a fake 'Comprovante' (payment receipt) page and exfiltrates the visitor's geolocation to an attacker-… | 24 | 1686 | experimental |
| paulpierre/informer A Python library built on Telethon that masquerades as real Telegram user accounts to monitor and log messages from hundreds of channels pe… | 50 | 1666 | experimental |
| SETA OWL is a Python‑based framework that enables multiple AI agents to collaborate on real‑world task automation. It builds on the CAMEL‑AI eco… | 60 | 179 | active |
| MapLibre maplibre-rs is a portable vector map renderer written in Rust that uses WebGPU for cross-platform rendering on web, mobile, and desktop. It… | 77 | 1584 | experimental |
| danielgross/LlamaAcademy LlamaAcademy is a Python pipeline that crawls API documentation, generates synthetic training data with GPT-3.5/GPT-4, and fine-tunes a Vic… | 30 | 1197 | experimental |
| kkyon/botflow Botflow is a Python dataflow programming framework for building data pipelines using pipes and routes, with parallelism via coroutines and … | 58 | 1193 | experimental |
| theacodes/kicanvas KiCanvas is an interactive, browser-based viewer for KiCad schematics and PCB boards, rendered with WebGL and the Canvas element. It is wri… | 64 | 1144 | experimental |
| distribyted/distribyted Distribyted is a torrent client written in Go that exposes torrent contents as standard filesystems via FUSE, WebDAV, or HTTP endpoints. It… | 60 | 1131 | experimental |
| oxylabs/ai-scraper-py AI-Scraper is a Python library and scrape agent from Oxylabs AI Studio that extracts data from webpages using natural language prompts inst… | 10 | 1090 | experimental |