Open-source web-scraping software
2000 open-source libraries, tools and applications tagged web-scraping, ranked by how closely web-scraping is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
2000 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| gautamkrishnar/socli SoCLI is a Python command-line client for Stack Overflow that lets developers search and browse questions and answers without leaving the t… | 52 | 1112 | active |
| SteamTracking/SteamTracking A project that tracks and reverse-engineers Steam and Valve data, including protobuf definitions and changes across Steam services. It moni… | 77 | 1110 | active |
| gxtrobot/bustag Bustag is a self-hosted web application that periodically scrapes new media title (番号) listings, lets users tag items as liked or disliked,… | 23 | 3817 | maintenance |
| shy1132/VacuumTube VacuumTube is an unofficial Electron-based desktop wrapper of YouTube Leanback, the official YouTube interface for consoles and Smart TVs. … | 83 | 1102 | active |
| Rain120/qq-music-api A QQ Music API service built with Koa2 and TypeScript that proxies web-side QQ Music endpoints for songs, artists, playlists, and rankings.… | 77 | 1102 | active |
| vifreefly/kimuraframework Kimuraframework (Kimurai) is a Ruby web scraping framework with an AI-assisted DSL: an LLM generates XPath selectors from a schema on first… | 57 | 1102 | active |
| Kappaemme-git/codex-first-customer-finder-skill A Codex skill (plugin) that takes a startup URL or product idea and produces an evidence-backed shortlist of potential first customers from… | 56 | 1100 | active |
| soxoj/socid-extractor socid_extractor is a Python library and CLI that extracts structured account metadata and stable internal identifiers (usernames, UIDs, GAI… | 88 | 1099 | active |
| kort0881/telegram-proxy-collector A Python CLI tool that automatically collects, analyzes, and filters MTProto and SOCKS5 proxies for Telegram. It decodes proxy secrets to d… | 79 | 1099 | active |
| requireCool/stealth.min.js A repository that automatically generates and publishes the newest stealth.min.js file every Monday at 6 AM UTC. The file is a drop-in Java… | 75 | 1098 | active |
| techtanic/Discounted-Udemy-Course-Enroller A Python application (with GUI and CLI variants) that scrapes websites for 100% off Udemy course coupons and automatically enrolls the user… | 65 | 1097 | active |
| Tuhinshubhra/RED_HAWK RED_HAWK is a PHP-based all-in-one reconnaissance and vulnerability scanning tool for websites. It performs information gathering (whois, D… | 32 | 3767 | maintenance |
| sharebook-kr/pykrx PyKrx is a Python library that scrapes stock and bond market data from the Korea Exchange (KRX) and Naver. It provides APIs for querying ti… | 81 | 1095 | active |
| xuejianxianzun/PixivFanboxDownloader A Chrome browser extension for batch downloading files from Pixiv Fanbox posts. It supports file type filtering, custom filenames, multiple… | 95 | 1094 | active |
| Casvt/Kapowarr Kapowarr is a self-hosted web application for building and managing a digital comic book library, designed to fit into the *arr suite of me… | 93 | 1093 | active |
| spider-ios/autox-release AutoX is a desktop social media operations tool that automates one-click publishing of videos to multiple platforms such as Douyin, TikTok,… | 60 | 1090 | active |
| Scrapfly A collection of educational Python web scraping scripts for over 40 popular domains such as Amazon, AliExpress, BestBuy, and Twitter, built… | 73 | 1088 | active |
| YaoZeyuan/stablog Stablog (稳部落) is a desktop application that backs up and exports a user's Weibo posts into searchable HTML and PDF ebooks. It logs into the… | 63 | 1088 | active |
| davis7dotsh/my-pi-setup An opinionated configuration and extension setup for the Pi coding agent, adding themes, background terminals, subagents, workflows, and se… | 59 | 1088 | active |
| jae-jae/fetcher-mcp A Model Context Protocol (MCP) server that fetches web page content using a Playwright headless browser, executing JavaScript to handle dyn… | 45 | 1088 | active |
| Tyrrrz/YoutubeExplode A .NET library providing an abstraction layer over YouTube's internal API to query metadata for videos, playlists, and channels, and to res… | 92 | 3735 | maintenance |
| 1061700625/WeChat_Article A PyQt5 desktop application that crawls and downloads all articles from a specified WeChat official account. It uses Selenium to log in and… | 58 | 1087 | active |
| nasa/apod-api NASA's open-source microservice that serves the Astronomy Picture of the Day (APOD) API, returning JSON metadata and image links parsed fro… | 66 | 1086 | active |
| GeneralMills/pytrends Pytrends is an unofficial Python library providing a pseudo API for Google Trends, enabling automated downloading of trend reports. It wrap… | 10 | 3726 | maintenance |
| Cloxl/xhshow A pure-algorithm Python library that generates Xiaohongshu (XHS/RedNote) request signature headers such as x-s, x-s-common, x-t, and x-rap-… | 70 | 1085 | active |
| dsclca12/auto_reg Any Auto Register is a self-hosted multi-platform account automatic registration and management system built with Python and Node.js. It in… | 49 | 1084 | active |
| HG-ha/ICP_Query A self-hosted service (Python and Rust implementations) that queries China MIIT ICP filing records for domains, apps, mini-programs, quick … | 89 | 1083 | active |
| Cinvin/myuserscripts A Tampermonkey userscript for NetEase Cloud Music's web player that adds song downloading, cloud-disk transfer, fast cloud-disk upload, and… | 72 | 1082 | active |
| ruipgil/scraperjs Scraperjs is a Node.js web scraping library offering two scrapers: a lightweight StaticScraper using cheerio for static HTML, and a Dynamic… | 32 | 3712 | maintenance |
| Whisparr Whisparr is an adult movie collection manager for Usenet and BitTorrent users, forked from the Sonarr/Radarr family. It monitors RSS feeds … | 95 | 1080 | active |
| Jikan Jikan is an unofficial PHP library and REST API for MyAnimeList.net that scrapes the website to provide data the official API lacks. It let… | 74 | 1080 | active |
| wreq wreq is an ergonomic, privacy-aware HTTP client written in Rust with a Python binding (wreq-python) that provides high-fidelity browser TLS… | 88 | 1077 | active |
| linkchecker/linkchecker LinkChecker is a GPL-licensed Python tool that checks links in web documents or entire websites for broken URLs. It supports recursive mult… | 66 | 1076 | active |
| m-sec-org/EZ EZ is a cross-platform vulnerability scanner that combines information gathering, port scanning, service brute-forcing, URL crawling, finge… | 24 | 1076 | active |
| viu-media/viu Viu is a terminal-based anime client that provides a rich TUI for browsing, searching, and managing your AniList library, along with stream… | 10 | 1076 | active |
| jmcarp/robobrowser RoboBrowser is a Pythonic library for browsing the web without a standalone browser, combining Requests for HTTP sessions with BeautifulSou… | 32 | 3688 | maintenance |
| borisbabic/browser_cookie3 A Python library (fork of browsercookie) that loads cookies from installed web browsers like Chrome, Firefox, Edge, Safari, and others into… | 23 | 1075 | active |
| tomnomnom/assetfinder A Go command-line tool that discovers domains and subdomains potentially related to a given domain by querying multiple passive sources lik… | 23 | 3682 | maintenance |
| tombcato/clash-ip-checker A Python automation tool for Clash proxy users that iterates through proxy nodes, checks each node's IP purity, bot ratio, and IP type via … | 38 | 1068 | active |
| chainreactors/spray Spray is a high-performance HTTP directory fuzzing and content discovery tool written in Go, positioned as a next-generation alternative to… | 87 | 1067 | active |
| Henryhaohao/Bilibili_video_download A Python tool for downloading videos from Bilibili, supporting single-part and multi-part (分P) videos, bangumi episodes, and multiple downl… | 32 | 3642 | maintenance |
| unitedstates/congress A community-run Python toolkit that collects and converts official U.S. Congress data—bills, amendments, roll call votes, nominations, and … | 49 | 1063 | active |
| jackwener/wechat-article-to-markdown A Python CLI tool that fetches WeChat Official Account articles using anti-detection browser automation (Camoufox) and converts them to cle… | 45 | 1063 | active |
| caolvchong-top/twitter_download A Python command-line tool that scrapes and downloads images, videos (including GIFs), and text from Twitter/X user timelines. It supports … | 71 | 1062 | active |
| cantino/selectorgadget SelectorGadget is an open-source bookmarklet and Chrome extension that generates CSS selectors for page elements through point-and-click se… | 66 | 1061 | active |
| Free Dictionary API A free REST API that returns dictionary data for English words, including definitions, phonetics, audio pronunciations, origins, synonyms, … | 32 | 3626 | maintenance |
| DowneyRem/PixivSource A Pixiv book source (plugin) for the Legado open-source reading app that turns it into a full-featured third-party Pixiv novel reader on An… | 93 | 1060 | active |
| QingJ01/123pan_unlock A Tampermonkey userscript that unlocks download restrictions on the 123pan (123云盘) cloud storage service, including bypassing the 1GB downl… | 10 | 1060 | active |
| TheNetsky/Microsoft-Rewards-Script A TypeScript bot built on Playwright that automates Microsoft Rewards (modern Bing Rewards dashboard) activities such as searches, daily se… | 90 | 1057 | active |
| Zarcolio/sitedorks A Python CLI tool that runs Google dork-style searches across multiple search engines (Google, Bing, DuckDuckGo, Yandex, Yahoo, Ecosia, Bra… | 73 | 1057 | active |
| lennybase/browsernode Browsernode is a TypeScript implementation of Browser-use that lets LLM-powered AI agents control a web browser via Playwright. It provides… | 31 | 1056 | active |
| robotshell/magicRecon MagicRecon is a Bash shell script that automates reconnaissance and vulnerability scanning of target domains, including subdomain enumerati… | 23 | 1056 | active |
| lm-rebooter/NuggetsBooklet A Node.js script that downloads Juejin (掘金) booklet content for personal study by reusing the user's authenticated browser cookies. It expo… | 62 | 1054 | active |
| Parse Video A Go library and CLI tool that parses short-video share links from 25+ Chinese platforms (Douyin, Kuaishou, Bilibili, Xiaohongshu, Weibo, e… | 74 | 1053 | active |
| 巨应壁纸 (LiveWallpaper) Giantapp Livewallpaper is an open-source wallpaper application for Windows 10/11 that supports both dynamic (video/animated) and static wal… | 88 | 1051 | active |
| TypNull/Tubifarry Tubifarry is a Lidarr plugin that adds extra music sources to your library management, using Spotify as an indexer and YouTube or Soulseek … | 85 | 1051 | active |
| mandatoryprogrammer/thermoptic Thermoptic is a stealth HTTP proxy that routes requests from any HTTP client (like curl) through a containerized Chrome instance so the tra… | 62 | 1051 | active |
| stay-leave/weibo-public-opinion-analysis A Python project for Weibo public opinion analysis that combines a web crawler, LDA topic modeling, sentiment analysis, and spatiotemporal … | 32 | 1051 | active |
| lijiejie/GitHack GitHack is a Python CLI exploit tool that reconstructs a website's source code from an exposed .git folder. It parses the .git/index file, … | 32 | 3581 | maintenance |
| kelvinBen/AppInfoScanner A Python-based static information-gathering scanner for mobile apps (Android APK/DEX, iOS IPA/Mach-O) and static web content (HTML, JS, H5)… | 99 | 3580 | maintenance |
| jaimeiniesta/metainspector MetaInspector is a Ruby gem for web scraping that fetches a given URL and exposes its title, meta description, keywords, links, images, cha… | 77 | 1049 | active |
| eHunter eHunter is a Tampermonkey/userscript that injects a Vue 3-based comic reader UI into supported comic sites (EH/EXHentai, NHentai), offering… | 70 | 1049 | active |
| turicas/brasil.io The backend of Brasil.IO, a platform that collects, cleans, and publishes Brazilian public open datasets in accessible formats. It automate… | 74 | 1047 | active |
| buffer/thug Thug is a Python low-interaction honeyclient that mimics the behavior of a web browser to detect and emulate malicious web content. It comp… | 92 | 1046 | active |
| maxzhang666/OneKeyVip A multi-function browser userscript (compatible with Tampermonkey and ScriptCat) that bundles VIP video/music parsing, Bilibili cover fetch… | 77 | 1046 | active |
| ma6254/FictionDown FictionDown is a Go-based command-line tool for batch downloading and crawling web novels from sites like Qidian and Biquge. It supports mu… | 23 | 1045 | active |
| TheAlgorithms/website The official website for The Algorithms, a static Next.js site that scrapes algorithm implementations from TheAlgorithms GitHub repositorie… | 76 | 1044 | active |
| webwhiz-ai/webwhiz WebWhiz is an open-source, self-hostable application that trains a ChatGPT-powered chatbot on your website data by crawling your pages and … | 45 | 1044 | active |
| Diving-Fish/maimaidx-prober A score tracker (prober) for the arcade rhythm game maimai DX that imports play records via a proxy tool and displays DX Rating and song sc… | 87 | 1042 | active |
| badlogic/heissepreise A self-hostable grocery price search app that daily scrapes product and price data from major Austrian supermarket chains (Billa, Spar, Hof… | 57 | 1041 | active |
| carcabot/tiktok-signature A self-hosted Node.js service that generates valid X-Bogus and X-Gnarly signature tokens for TikTok API requests using a headless browser r… | 87 | 1040 | active |
| Anorov/cloudflare-scrape A Python module (cfscrape) built on Requests that bypasses Cloudflare's JavaScript anti-bot challenge page ('I'm Under Attack Mode') so scr… | 23 | 3542 | maintenance |
| igrigorik/ga-beacon A small Go service that acts as a collector-as-a-service for Google Analytics via the Measurement Protocol, letting you track page views wi… | 32 | 3536 | maintenance |
| TheRook/subbrute SubBrute is a Python DNS meta-query spider that enumerates subdomains and arbitrary DNS record types by leveraging open resolvers to bypass… | 23 | 3535 | maintenance |
| JosephLai241/URS URS (Universal Reddit Scraper) is a comprehensive command-line tool written in Python (with Rust components) for scraping and archiving Red… | 60 | 1037 | active |
| fossology/fossology FOSSology is an open source license compliance system and toolkit that scans software for licenses, copyrights, and export control data. It… | 90 | 1035 | active |
| arabcoders/ytptube YTPTube is a self-hosted web-based download manager and automation layer for yt-dlp. It combines scheduled tasks, metadata-driven condition… | 88 | 1033 | active |
| Vinyzu/Botright Botright is a Python browser automation framework built on Playwright that provides undetectable, fingerprint-changing stealth browsing. It… | 75 | 1033 | active |
| instagram4j/instagram4j instagram4j is an object-oriented, reverse-engineered Instagram Private API client library for Java. It lets developers log in, post, messa… | 74 | 1033 | active |
| am-will/codex-skills A collection of Codex/agent skills written in Shell covering planning, documentation access, prompting, frontend design guidance, Codex too… | 54 | 1033 | active |
| EdgeSecurityTeam/EHole EHole (棱洞) is a Go-based fingerprint identification tool for red team reconnaissance that pinpoints high-value, easily attackable systems (… | 23 | 3515 | maintenance |
| agregarr/agregarr Agregarr is a self-hosted, Docker-based Plex Collections manager that automatically creates and refreshes collections from sources like Tra… | 61 | 1031 | active |
| Wikidepia/InstaFix InstaFix is a Go web service that serves fixed Instagram image and video embeds for Discord and Telegram by rewriting URLs (e.g., adding 'd… | 10 | 1030 | active |
| daijro/hrequests hrequests is a Python HTTP client library that replaces the requests library with browser TLS fingerprint replication, HTTP/2 support, and … | 31 | 1026 | active |
| hrithikkoduri/WebRover WebRover is an autonomous AI web agent that interprets user input, navigates websites via browser automation, and performs tasks or deep re… | 12 | 1026 | active |
| wnma3mz/wechat_articles_spider A Python library for scraping WeChat Official Account articles, including article URLs, reading counts, likes, and comments. It can also do… | 23 | 3481 | maintenance |
| mxrch/GitFive GitFive is a Python-based OSINT CLI tool for investigating GitHub user profiles. It uncovers usernames, name history, email addresses, and … | 39 | 1019 | active |
| owner888/phpspider phpspider is a PHP web crawling framework that lets developers build scrapers with a simple config array, handling multi-process workers, l… | 65 | 3459 | maintenance |
| JoMingyu/google-play-scraper A Python library that provides APIs to crawl the Google Play Store for app details, reviews, and other data without any external dependenci… | 32 | 1018 | active |
| ranahaani/GNews GNews is a lightweight Python package that queries the Google News RSS feed and returns article results as usable JSON. It supports keyword… | 66 | 1013 | active |
| vimcolorschemes/vimcolorschemes A website and open-source app for browsing and discovering Vim and Neovim colorschemes from GitHub repositories. It tracks thousands of rep… | 94 | 1011 | active |
| vasani-arpit/WBOT WBOT is a Node.js-based bot for WhatsApp Web that automates message replies using Puppeteer to control a browser. It is configurable via a … | 50 | 1010 | active |
| pea3nut/Pxer Pxer is a userscript (installed via Tampermonkey) that acts as a crawler for pixiv.net, letting users batch-fetch artworks, collections, an… | 23 | 1010 | active |
| stitionai/devika Devika is an open‑source autonomous AI software engineer that can interpret high‑level instructions, research, plan, and write code across … | 41 | 19559 | experimental |
| ViggoZ/producthunt-daily-hot A Python automation tool that generates a daily Chinese-language digest of Product Hunt's top 30 products, using the Product Hunt API and O… | 69 | 1005 | active |
| ki9mu/ARL-plus-docker A Docker-based fork of ARL (Asset Reconnaissance Lighthouse) v2.6.2 that performs automated asset discovery and vulnerability scanning for … | 32 | 1000 | active |
| gwen001/pentest-tools A collection of small custom security scripts in Bash, Python, and PHP for penetration testing and bug bounty quick tasks, covering DNS enu… | 67 | 3342 | maintenance |
| kevinzg/facebook-scraper A Python library for scraping public Facebook pages, groups, profiles, and posts without requiring an API key. It provides a simple get_pos… | 32 | 3313 | maintenance |
| Instagram Private API A Python client library for Instagram's private (undocumented) app and web APIs, with no third-party dependencies. It exposes app-only feat… | 23 | 3311 | maintenance |
| s-rah/onionscan OnionScan is a free and open source Go CLI tool for investigating Tor hidden services (.onion sites) on the Dark Web. It scans sites for op… | 23 | 3306 | maintenance |