Open-source web-scraping software
2000 open-source libraries, tools and applications tagged web-scraping, ranked by how closely web-scraping is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
2000 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| alixaxel/chrome-aws-lambda A TypeScript library that ships a size-optimized Chromium binary designed to run headless browser automation with Puppeteer (or Playwright)… | 32 | 3284 | maintenance |
| archivy/archivy Archivy is a self-hostable personal knowledge repository that combines note-taking, bookmarking with full-page preservation, and a searchab… | 23 | 3280 | maintenance |
| cztomczak/cefpython CEF Python provides Python bindings for the Chromium Embedded Framework, letting Python applications embed a full Chromium-based browser. I… | 67 | 3237 | maintenance |
| wenbochang888/house A Java web scraper built with SpringBoot, HttpClient, and JSoup that crawls famous Tianya forum threads about China's housing market and co… | 32 | 3227 | maintenance |
| scrapy-plugins/scrapy-splash A Scrapy plugin that integrates the Splash headless browser service to enable crawling and scraping of JavaScript-rendered web pages. It pr… | 23 | 3225 | maintenance |
| Hawk Hawk is a visual crawler and ETL IDE written in C#/WPF that lets users graphically scrape webpages, clean, transform, and store data withou… | 23 | 3205 | maintenance |
| atlas-comstock/NeteaseCloudMusicFlac A Python command-line script that downloads lossless FLAC music files to local storage based on a Netease Cloud Music playlist URL. It auto… | 32 | 3150 | maintenance |
| tomnomnom/httprobe httprobe is a Go command-line tool that takes a list of domains on stdin and probes for working HTTP and HTTPS servers, reporting which res… | 23 | 3126 | maintenance |
| 0x0be/yesitsme A Python CLI script for OSINT investigations that finds Instagram profiles matching a given name, e-mail, or phone number. It scrapes dumpo… | 32 | 3087 | maintenance |
| CrawlScript/WebCollector WebCollector is an open-source Java web crawler framework that provides simple interfaces for building multi-threaded web crawlers quickly.… | 58 | 3081 | maintenance |
| hongchacha/cartoon An Android app dedicated to reading comics and manga, aggregating content from dozens of Chinese comic websites. It replaces a web browser … | 32 | 3011 | maintenance |
| CodeRayZhang/Movie_Recommend A full-stack movie recommendation system built on Spark, including a Scrapy crawler, an SSM-based movie website, an admin backend, and a Sp… | 32 | 3008 | maintenance |
| jaeles-project/gospider GoSpider is a fast web spider/crawler written in Go that crawls sites in parallel and extracts URLs from sitemaps, robots.txt, JavaScript f… | 23 | 3008 | maintenance |
| mathieudutour/medium-to-own-blog A CLI tool that migrates a Medium blog to a self-hosted Gatsby-based blog in minutes. It exports Medium content into Markdown/MDX and scaff… | 32 | 3000 | maintenance |
| kotartemiy/newscatcher A Python package that programmatically collects normalized news articles from thousands of news websites, filterable by topic, country, and… | 32 | 2988 | maintenance |
| x0rz/tweets_analyzer A Python command-line script that scrapes a Twitter profile's tweets and analyzes metadata such as activity patterns, timezone, sources, ge… | 23 | 2982 | maintenance |
| google-play-scraper A Node.js library that scrapes application data from the Google Play store, exposing methods for app details, search, reviews, permissions,… | 67 | 2981 | maintenance |
| JSFinder JSFinder is a Python command-line tool that crawls a website's JavaScript files and extracts URLs and subdomains using regex parsing. It su… | 32 | 2975 | maintenance |
| iptv-m3u-maker A Python tool that scrapes publicly shared IPTV live-stream sources, tests each link's latency on the local network, and generates an optim… | 32 | 2896 | maintenance |
| howie6879/owllook owllook is a self-hosted vertical search engine for Chinese web novels, built on Python with Sanic, MongoDB, and Redis. It aggregates resul… | 77 | 2889 | maintenance |
| adieuadieu/serverless-chrome Serverless Chrome is a scaffolding framework and Serverless Framework plugin for running headless Chrome/Chromium on AWS Lambda. It bundles… | 23 | 2888 | maintenance |
| CharlesPikachu/DecryptLogin A Python library providing programmatic login APIs for popular websites (Weibo, Bilibili, Zhihu, GitHub, Taobao, etc.) built on the request… | 32 | 2853 | maintenance |
| bhavsec/reconspider ReconSpider is an open-source OSINT framework written in Python for scanning IP addresses, emails, websites, and organizations to gather in… | 23 | 2827 | maintenance |
| ganlvtech/down_52pojie_cn A single-page file explorer built with Vue.js that can be hosted on any static website, originally created to power the 52pojie forum's dow… | 23 | 2814 | maintenance |
| martinvigo/email2phonenumber A Python OSINT tool that discovers a target's phone number from just their email address by abusing password reset flows that leak masked p… | 32 | 2808 | maintenance |
| hello-efficiency-inc/raven-reader Raven Reader is an open-source cross-platform desktop RSS/news reader built with Electron and Vue that aggregates articles from feeds in a … | 10 | 2807 | maintenance |
| topfunky/hpple Hpple is an Objective-C wrapper around the XPathQuery libxml2 library for parsing HTML and searching it with XPath expressions. It was insp… | 32 | 2794 | maintenance |
| yann-shi/dht A Go library implementing the BitTorrent DHT protocol (BEP-3, 5, 9, 10) with two modes: a standard DHT server and a crawling mode for harve… | 23 | 2771 | maintenance |
| DormyMo/SpiderKeeper SpiderKeeper is a self-hosted web-based admin dashboard for managing Scrapy spiders running on Scrapyd servers. It provides spider scheduli… | 32 | 2764 | maintenance |
| Xmader/musescore-downloader A tool for downloading sheet music from musescore.com for free without a login or Musescore Pro subscription, available as a CLI (npx msdl)… | 23 | 2764 | maintenance |
| brianway/webporter webporter is a Java crawler application built on the webmagic framework that demonstrates a complete pipeline of data crawling, persistence… | 23 | 2764 | maintenance |
| jeanphix/Ghost.py Ghost.py is a scriptable WebKit-based web client library for Python, built on PySide2/Qt5, allowing programmatic page loading and content i… | 32 | 2752 | maintenance |
| jerry3747/taobao_seckill A Python Selenium-based script that automates flash-sale purchases on Taobao and Tmall, timing checkout to grab discounted items like TVs o… | 32 | 2694 | maintenance |
| pocketjoso/penthouse Penthouse is a Node.js library that generates critical path CSS for web pages, extracting the CSS needed to render above-the-fold content f… | 23 | 2679 | maintenance |
| weskerfoot/DeleteFB A Python CLI tool that uses Selenium to automate deleting your Facebook posts, page likes, and conversations through your real browser. It … | 10 | 2662 | maintenance |
| mckaywrigley/paul-graham-gpt An open-source AI-powered search and chat application built over Paul Graham's essays using OpenAI embeddings, pgvector on Supabase, and GP… | 31 | 2661 | maintenance |
| thewhiteh4t/nexfil Nexfil is an OSINT command-line tool written in Python that finds social media profiles by username across 350+ websites in seconds. It sup… | 32 | 2630 | maintenance |
| mattt/Ono Ono is an Objective-C/Swift library for parsing and querying XML and HTML documents on iOS and macOS, built on libxml2. It provides a conve… | 32 | 2591 | maintenance |
| obheda12/GitDorker GitDorker is a Python CLI tool that uses the GitHub Search API with a curated list of over 200 dorks to find sensitive information exposed … | 32 | 2589 | maintenance |
| Ccixyj/JBusDriver An Android app for browsing the JAVBUS adult content catalog, inspired by JAViewer. It is built with Kotlin using MVP architecture, RxJava2… | 23 | 2585 | maintenance |
| Tuhinshubhra/CMSeeK CMSeeK is a Python 3 command-line suite that detects the content management system (CMS) powering a website, supporting over 180 CMSs inclu… | 61 | 2578 | maintenance |
| justinzm/gopup GoPUP is a Python library that provides convenient interfaces to a wide range of public Chinese data sources, including Baidu/Weibo/Google … | 23 | 2545 | maintenance |
| luin/readability A Node.js library that extracts clean, readable article content from any web page, based on arc90's readability project. It returns the art… | 32 | 2517 | maintenance |
| Gecco Gecco is a lightweight, easy-to-use web crawler framework for Java that lets developers define crawlers with annotation-based jQuery-style … | 48 | 2513 | maintenance |
| AIMedia AIMedia is a heavyweight integrated application that automatically scrapes trending news, generates AI-written articles with AI-generated i… | 66 | 2484 | maintenance |
| yichahucha/surge A collection of JavaScript rewrite scripts for the Surge and Quantumult X proxy apps. They modify HTTP responses on iOS to inject Netflix I… | 66 | 2467 | maintenance |
| lorien/grab Grab is a Python web scraping framework providing HTTP request handling, proxy/cookie support, and XPath-based HTML parsing, plus a Spider … | 48 | 2463 | maintenance |
| andyzys/jd_seckill A Python script that automates logging into JD.com (Jingdong), reserving products like Moutai liquor, and purchasing them at a scheduled se… | 32 | 2448 | maintenance |
| decaywood/XueQiuSuperSpider A Java 8 web scraping framework for collecting stock data from Xueqiu (Snowball) and other Chinese financial sites. It is built around comp… | 32 | 2439 | maintenance |
| screetsec/Sudomy Sudomy is a Bash-based subdomain enumeration and reconnaissance framework that collects subdomains via active brute-forcing and passive thi… | 23 | 2434 | maintenance |
| netlify/staticgen StaticGen.com is a Gatsby-built leaderboard website that ranks open-source static site generators by GitHub/GitLab stars, forks, and Twitte… | 10 | 2432 | maintenance |
| delight-im/Android-AdvancedWebView An enhanced WebView component library for Android that fixes common WebView quirks and works out of the box. It provides a drop-in replacem… | 32 | 2412 | maintenance |
| paquettg/php-html-parser A PHP library that parses HTML into a DOM and lets you find and manipulate tags using CSS selectors, similar to jQuery. It is designed for … | 32 | 2395 | maintenance |
| Terminus2049/Terminus2049.github.io Terminus (端点星) is a GitHub Pages-hosted site built with Jekyll that archives articles deleted from Chinese platforms like WeChat and Weibo … | 32 | 2372 | maintenance |
| QianyanTech/Image-Downloader A Python application that crawls and downloads images from Google, Bing, and Baidu using Selenium or API drivers. It offers both a PyQt5 GU… | 23 | 2359 | maintenance |
| madelynnblue/goread Go Read is a Google Reader clone: a self-hostable RSS feed reader built in Go on Google App Engine with an AngularJS frontend. It lets user… | 10 | 2358 | maintenance |
| Tangram Tangram is a JavaScript library for rendering 2D and 3D maps live in the browser using WebGL, tuned for OpenStreetMap but supporting GeoJSO… | 49 | 2334 | maintenance |
| hidroh/materialistic Materialistic is a material-design Hacker News client for Android built in Java. It uses the official Hacker News API, Algolia search, and … | 23 | 2331 | maintenance |
| Encom Boardroom A WebGL/Three.js web application recreating the Encom boardroom scene from Tron: Legacy, visualizing realtime GitHub and Wikipedia data str… | 32 | 2320 | maintenance |
| lucasjinreal/weibo_terminater A Python-based web scraper that crawls Weibo (Sina's microblog platform) to collect user posts, comments, followers, and conversation pairs… | 32 | 2317 | maintenance |
| ggerganov/wave-share A proof-of-concept browser application that shares files between nearby devices by performing WebRTC signaling through audio tones instead … | 32 | 2313 | maintenance |
| sfu-db/dataprep DataPrep is a Python library for low-code data preparation, offering modules to collect data from common APIs (connector), run fast explora… | 23 | 2261 | maintenance |
| buckket/twtxt twtxt is a decentralised, minimalist microblogging tool and format specification where your identity is a publicly accessible text file of … | 23 | 2235 | maintenance |
| j0be/PowerDeleteSuite A browser bookmarklet for Reddit that lets users mass-edit and mass-delete their own comments and submissions using Reddit's API endpoints.… | 23 | 2187 | maintenance |
| jadepeng/XMusicDownloader A C# desktop application that aggregates music search across Baidu, NetEase, QQ, Kugou, and Migu music sites and supports batch downloading… | 23 | 2174 | maintenance |
| ageitgey/node-unfluff A Node.js library and CLI tool that automatically extracts the main body content and metadata (title, author, date, images, tags, links) fr… | 32 | 2160 | maintenance |
| t9tio/cloudquery CloudQuery is a tool that turns any website into a JSON API by fetching pages with headless Chrome and extracting data via CSS selectors. I… | 32 | 2149 | maintenance |
| anouarbensaad/vulnx VulnX is a Python CLI tool that detects CMS types (WordPress, Joomla, Drupal, etc.), gathers target information like subdomains and DNS rec… | 23 | 2149 | maintenance |
| initstring/cloud_enum A Python command-line OSINT tool that enumerates publicly exposed resources across AWS, Azure, and Google Cloud using keyword mutations and… | 90 | 2148 | maintenance |
| D4Vinci/Cr3dOv3r Cr3dOv3r is a Python command-line pentesting tool for investigating credential reuse attacks. Given an email, it searches public breach dat… | 54 | 2144 | maintenance |
| php-embed/Embed A PHP library that extracts metadata and embed information from any web page or web service using oEmbed, OpenGraph, Twitter Cards, and HTM… | 87 | 2141 | maintenance |
| UnaPibaGeek/ctfr CTFR is a Python command-line tool that enumerates HTTPS website subdomains by querying Certificate Transparency logs (via crt.sh) instead … | 32 | 2126 | maintenance |
| cyberagiinc/DevDocs DevDocs is a free, private, UI-based MCP server that crawls and extracts technical documentation (using Crawl4AI and Playwright) and expose… | 47 | 2108 | maintenance |
| TideSec/WDScanner WDScanner is a self-hosted distributed web vulnerability scanning platform written in PHP with Python backend workers. It combines customer… | 32 | 2099 | maintenance |
| Aabyss-Team/ARL ARL (Asset Reconnaissance Lighthouse) is a self-hosted asset reconnaissance system that quickly discovers internet-facing assets associated… | 26 | 2087 | maintenance |
| stevenvachon/broken-link-checker A Node.js library and CLI tool (blc) that crawls HTML to find broken links, missing images, and other dead URLs. It supports concurrent, st… | 74 | 2081 | maintenance |
| iSafeBlue/TrackRay TrackRay (溯光) is an open-source penetration testing framework written in Java on SpringBoot that implements its own vulnerability scanning … | 23 | 2079 | maintenance |
| vim-awesome/vim-awesome Vim Awesome is an open-source web application that serves as a comprehensive directory of Vim plugins, aggregating data from GitHub, Vim.or… | 23 | 2063 | maintenance |
| PuerkitoBio/gocrawl gocrawl is a polite, slim and concurrent web crawler library written in Go. It respects robots.txt rules, applies per-host crawl delays, an… | 23 | 2051 | maintenance |
| 670848654/SakuraAnime A third-party Android client for the anime streaming sites Yhdm (Sakura Anime) and SiliSili, built in Java using jsoup for scraping site co… | 10 | 2041 | maintenance |
| 0xbug/Hawkeye Hawkeye is a self-hosted system that monitors GitHub for leaked sensitive information, such as employees pushing company code or credential… | 23 | 2037 | maintenance |
| Nekmo/dirhunt Dirhunt is a Python CLI web crawler optimized for finding and analyzing web directories without brute-forcing paths. It detects 'index of' … | 23 | 2006 | maintenance |
| awolfly9/IPProxyTool A Python/Scrapy application that crawls free proxy websites, validates the collected proxy IPs against target sites, and stores usable prox… | 32 | 2004 | maintenance |
| bigemon/ChatGPT-ToolBox A ChatGPT toolbox userscript/browser extension written largely by ChatGPT itself, injectable via Tampermonkey or Chrome bookmarklets. It ad… | 32 | 2001 | maintenance |
| vaguileradiaz/tinfoleak tinfoleak is an open-source Python tool for OSINT/SOCMINT analysis of Twitter accounts, extracting structured intelligence such as user act… | 32 | 1986 | maintenance |
| youusername/magnetX magnetX is a native macOS application written in Objective-C for searching magnet links across multiple torrent search sites without browse… | 23 | 1985 | maintenance |
| microsoft/magentic-ui MagenticLite is an experimental agentic application from Microsoft AI Frontiers that automates tasks across the browser and local file syst… | 78 | 10097 | experimental |
| D35m0nd142/LFISuite LFISuite is a fully automatic Python tool that scans for and exploits Local File Inclusion (LFI) vulnerabilities using eight different atta… | 23 | 1972 | maintenance |
| coursera-dl/edx-dl A Python command-line tool that downloads video lectures and course materials from Open edX-based platforms such as edX.org, Stanford Onlin… | 23 | 1962 | maintenance |
| SathyaBhat/spotify-dl A Python CLI tool that fetches track metadata from Spotify playlists, albums, or tracks via the Spotify Web API and downloads the correspon… | 60 | 1950 | maintenance |
| Xyntax/POC-T POC-T is a Python 2.7 plugin-based concurrent framework for penetration testing tasks such as crawling, bruteforcing, and batch PoC/EXP ver… | 23 | 1938 | maintenance |
| stevenyomi/copymanga An unofficial Tachiyomi extension that adds CopyManga (拷贝漫画) as a manga reading source. It is installed as a trusted extension within the T… | 10 | 1918 | maintenance |
| bugswriter/notflix Notflix is a shell script that scrapes 1337x for magnet links and streams the resulting torrents via peerflix. It is a lightweight command-… | 32 | 1901 | maintenance |
| scrapy/scrapely Scrapely is a pure-Python library for extracting structured data from HTML pages. It learns a parser from example pages annotated with the … | 32 | 1883 | maintenance |
| eldraco/domain_analyzer Domain Analyzer is a Python-based security analysis tool that automatically discovers and reports information about a given domain, includi… | 32 | 1863 | maintenance |
| kanishka-linux/reminiscence Reminiscence is a self-hosted bookmark and archive manager built with Django that saves web pages in HTML, PDF, or full-page PNG formats. I… | 23 | 1859 | maintenance |
| alvarobartt/investpy investpy is a Python package for retrieving recent and historical financial data from Investing.com, covering stocks, funds, ETFs, indices,… | 54 | 1858 | maintenance |
| awake1t/linglong Linglong is a self-hosted asset reconnaissance and scanning system written in Go that continuously discovers network assets using masscan+n… | 32 | 1851 | maintenance |
| Lingva Translate Lingva Translate is a free and open-source alternative front-end for Google Translate that supports over a hundred languages. It scrapes Go… | 32 | 1848 | maintenance |
| xianhu/PSpider PSpider is a simple, easy-to-read web spider framework written in Python 3.8+. It uses a Fetcher/Parser/Saver pipeline with queues for mult… | 32 | 1833 | maintenance |