Open-source web-scraping software
2000 open-source libraries, tools and applications tagged web-scraping, ranked by how closely web-scraping is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
2000 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| towfiqi/serpbear SerpBear is an open-source, self-hosted search engine position tracking app for monitoring website keyword rankings in Google. It scrapes S… | 77 | 2096 | active |
| zorlan/skycaiji SkyCaiji (蓝天采集器) is an open-source, PHP+MySQL based visual web scraping system where users define collection rules by point-and-click in a … | 75 | 2095 | active |
| Oxylabs Amazon Scraper A Python-based example repository and free CLI tool for scraping Amazon product prices, best sellers, search results, and deals from depart… | 62 | 2093 | active |
| probberechts/soccerdata A Python library of scrapers that collect soccer data from popular websites like FBref, ESPN, WhoScored, Sofascore, SoFIFA, Understat, Club… | 89 | 2088 | active |
| ChinaGodMan/UserScripts A collection of Tampermonkey/Greasyfork userscripts modified from the internet, written in JavaScript. The scripts add browser enhancements… | 69 | 2088 | active |
| TrianguloY/URLCheck URLCheck is an open-source Android app that acts as an intermediary when opening URLs, letting users inspect, clean, and modify links befor… | 83 | 2082 | active |
| LinkeDOM LinkeDOM is a triple-linked-list based DOM implementation for DOM-less environments like Node.js and Deno, closely following the DOM standa… | 70 | 2080 | active |
| A9T9/RPA Ui.Vision RPA is an open-source robotic process automation tool delivered as a browser extension for Chrome, Edge, and Firefox, compatible … | 94 | 2064 | active |
| upbit/pixivpy PixivPy is a Python client library for the Pixiv App-API, supporting authenticated access via refresh tokens. It provides methods for searc… | 34 | 2062 | active |
| vmoranv/jshookmcp An MCP server exposing 600+ tools across 34 domains for JavaScript reverse engineering and security research, including browser automation,… | 72 | 2056 | active |
| rubycdp/ferrum Ferrum is a Ruby library providing a clean, high-level API to control Chrome or Chromium via the Chrome DevTools Protocol (CDP), with no Se… | 85 | 2055 | active |
| wbt5/real-url A Python collection of scripts that extracts real streaming URLs (live stream sources) and danmaku (bullet comments) from 59 Chinese and in… | 32 | 7837 | maintenance |
| nottelabs/notte Notte is a full-stack framework and cloud platform for building, deploying, and scaling AI web agents and browser automations. It combines … | 85 | 2040 | active |
| Field Theory A TypeScript CLI that syncs X/Twitter bookmarks to local markdown files, provides BM25 full-text search and LLM-based classification, and e… | 57 | 2036 | active |
| pt-plugins/PT-Plugin-Plus PT-Plugin-Plus is a Web Extensions browser plugin for Chrome, Edge, and Firefox that streamlines using private tracker (PT) sites, enabling… | 10 | 7799 | maintenance |
| karpathy/jobs A research tool that scrapes the Bureau of Labor Statistics Occupational Outlook Handbook (342 occupations) and renders an interactive tree… | 44 | 2029 | active |
| john-kurkowski/tldextract A Python library that accurately splits URLs into subdomain, domain, and public suffix components using the Public Suffix List. It also shi… | 90 | 2027 | stable |
| jonhoo/fantoccini Fantoccini is a Rust library providing a high-level async API for programmatically controlling browsers via the WebDriver protocol. It supp… | 76 | 2027 | active |
| jimmc414/onefilellm OneFileLLM is a Python command-line tool and library that aggregates content from sources like GitHub repos, pull requests, arXiv/Sci-Hub p… | 66 | 2022 | active |
| AnswerOverflow/AnswerOverflow Answer Overflow is an open-source application that indexes Discord server threads into searchable, SEO-friendly web pages so community know… | 66 | 2020 | active |
| elliotgao2/gain Gain is an asynchronous web crawling framework for Python built on asyncio, aiohttp, and lxml/pyquery. Users declare items and parsers decl… | 72 | 2019 | active |
| supermemoryai/markdowner Markdowner is a fast web service that converts any website into LLM-ready markdown, with optional LLM filtering, detailed responses, and au… | 25 | 2019 | active |
| ssili126/tv A Python tool that automatically collects IPv4 hotel IPTV live stream sources (including CCTV, satellite, and some local Chinese channels),… | 71 | 1997 | active |
| ZeroPointSix/outlookEmailPlus OutlookMail Plus is a self-hosted email manager purpose-built for account registration and verification workflows. It automates fetching ve… | 76 | 1996 | active |
| tsingyuai/growth-lab Growth Lab is an open-source, end-to-end AI growth system that runs product marketing and user-acquisition loops through natural-language c… | 57 | 1991 | active |
| zhegexiaohuozi/SeimiCrawler SeimiCrawler is an agile, standalone, distributed Java crawler framework inspired by Python's Scrapy, with deep Spring Boot integration and… | 84 | 1986 | active |
| Flexget/Flexget FlexGet is a multipurpose Python automation tool for content like torrents, NZBs, podcasts, comics, series, and movies. It pulls from sourc… | 95 | 1984 | active |
| 404-novel-project/novel-downloader An extensible userscript (Tampermonkey/Greasemonkey/Violentmonkey) that downloads novels from many Chinese web novel sites and exports them… | 77 | 1980 | active |
| KEV0143/Parser-Chitai-Gorod A Python-based scraper for the Russian online bookstore Chitai-Gorod that collects book URLs across catalog pages and extracts structured p… | 72 | 1980 | active |
| tiantianGPU/reg-factory A Python-based local web console that automates bulk registration of email and AI service accounts (Outlook, Gmail, ChatGPT, Grok, Claude, … | 78 | 1979 | active |
| ViennaRSS/vienna-rss Vienna is a free, open-source RSS/Atom/JSON feed newsreader for macOS with a native Apple Mail-like interface. It can fetch feeds directly … | 94 | 1974 | active |
| danmactough/node-feedparser A Node.js library for parsing RSS, Atom, and RDF syndication feeds as a streaming interface. It resolves relative URLs and correctly handle… | 66 | 1973 | stable |
| ThePhaseless/Byparr Byparr is a self-hosted Python service that solves antibot browser challenges (like Cloudflare checks) and returns valid clearance cookies … | 87 | 1966 | active |
| Tabula Tabula is a local web application for extracting data tables from text-based PDF files into CSV, Excel, or JSON. It is powered by the tabul… | 24 | 7488 | maintenance |
| autoclaw-cc/xiaohongshu-skills A set of AI agent skills (SKILL.md format) plus a Chrome extension that automates Xiaohongshu (RED) using your real logged-in browser sessi… | 64 | 1938 | active |
| SleepingBag945/dddd dddd is a Go-based batch information gathering and supply-chain vulnerability detection CLI tool designed to streamline red team workflows.… | 19 | 1937 | active |
| aeonfun/opendia OpenDia is an open-source browser extension (npm package) that connects your Chrome, Arc, or Firefox browser to AI models via the Model Con… | 86 | 1927 | active |
| damoeb/rss-proxy RSS-proxy is a self-hostable web service that generates RSS, ATOM, or JSON feeds from almost any static website by analyzing its HTML struc… | 32 | 1924 | active |
| linbailo/zyqinglong A collection of self-use scripts for the Qinglong panel that automates daily check-ins and coupon collection for Chinese apps like Didi, Me… | 37 | 1922 | active |
| Jesseovo/last30days-skill-cn An AI Agent skill (for Claude Code / OpenClaw) that automatically searches content from the last 30 days across 8 major Chinese internet pl… | 59 | 1916 | active |
| extractus/article-extractor A TypeScript library that extracts the main article content, title, image, and metadata from a given URL or raw HTML string. It supports cu… | 88 | 1913 | active |
| xtekky/TikTok-ViewBot A Python-based TikTok view bot that sends fake views via HTTP requests against Zefoy, without needing Selenium. It includes an automatic ca… | 75 | 1913 | active |
| sjdonado/idonthavespotify A web app that converts music links between streaming services like Spotify, Apple Music, YouTube Music, Tidal, and Deezer. It parses the s… | 92 | 1910 | active |
| karlicoss/promnesia Promnesia is a browser extension plus Python backend that enhances browsing history by annotating visited pages with context from multiple … | 79 | 1908 | active |
| pmh1314520/WebRPA WebRPA is an open-source, no-code visual RPA tool for building automation workflows by dragging and connecting modules, covering web scrapi… | 82 | 1904 | active |
| zema1/watchvuln WatchVuln is a self-hosted Go service that scrapes high-quality vulnerability sources (Aliyun AVD, Chaitin, OSCS, Qianxin TI, Seebug, CISA … | 78 | 1897 | active |
| bellingcat/octosuite Octosuite is a terminal-based toolkit for analyzing GitHub data, usable as an interactive TUI, a CLI, or a Python library. It queries user,… | 85 | 1895 | active |
| trevorhobenshield/twitter-api-client A Python library implementing X/Twitter's v1, v2, and GraphQL APIs for automation and scraping. It supports account actions like tweeting, … | 31 | 1895 | active |
| nsonaniya2010/SubDomainizer SubDomainizer is a Python CLI tool that discovers hidden subdomains and secrets in webpages, external JavaScript files, GitHub, and local f… | 65 | 1893 | active |
| ARC-MX/sgcc_electricity_new A Home Assistant integration (deployed via Docker) that scrapes China State Grid (SGCC) accounts to fetch electricity billing and usage dat… | 83 | 1892 | active |
| joyce677/TrendRadar TrendRadar is a lightweight self-hosted hot-topic monitor that aggregates trending news from 35 Chinese platforms (Weibo, Zhihu, Bilibili, … | 63 | 1892 | active |
| coder-hxl/x-crawl x-crawl is a flexible Node.js crawler library that supports crawling dynamic pages, static pages, API data, and files, with optional AI ass… | 67 | 1889 | active |
| Yōkai Yōkai is a free and open source manga reader app for Android, forked from Tachiyomi/Mihon. It supports local and online manga reading with … | 90 | 1886 | active |
| sarperavci/GoogleRecaptchaBypass A Python library that automatically solves Google reCAPTCHA v2 challenges in under five seconds using browser automation with DrissionPage … | 65 | 1883 | active |
| scholarly-python-package/scholarly scholarly is a Python library for retrieving author and publication metadata from Google Scholar through a friendly, Pythonic API. It handl… | 52 | 1880 | active |
| download-directory/download-directory.github.io A web app that lets users download a single subdirectory from a GitHub repository as a zip file, filling a gap in GitHub's native functiona… | 66 | 1878 | active |
| wapiti-scanner/wapiti Wapiti is an open-source black-box web vulnerability scanner written in Python that crawls deployed web applications and fuzzes scripts and… | 95 | 1877 | active |
| initstring/linkedin2username A Python OSINT tool that scrapes LinkedIn employee lists for a target company and generates multiple probable username formats (e.g., first… | 70 | 1876 | active |
| inulute/medium-unlocker Medium Unlocker is an Android app (with an accompanying web frontend) that bypasses Medium's paywall by redirecting article URLs to the fre… | 92 | 1870 | active |
| nasa-gibs/worldview NASA Worldview is an interactive web application for browsing over 1000 global, full-resolution satellite imagery layers from NASA's Global… | 92 | 1869 | active |
| 1234567Yang/cf-proxy-ex A Cloudflare Workers-based super proxy that lets users access websites like GitHub, DuckDuckGo, and StackOverflow through a different URL w… | 74 | 1865 | active |
| LagradOst/QuickNovel QuickNovel is a free, ad-free, open-source Android app for downloading novels from many web novel sites, which also functions as an EPUB re… | 98 | 1864 | active |
| ShunCai/QZoneExport A browser extension (Chrome/Edge/Firefox, Manifest V3) that backs up QQ Zone data—posts, blogs, private diaries, albums, videos, comments, … | 84 | 1863 | active |
| 0xKayala/NucleiFuzzer NucleiFuzzer is a Python-based automation tool that combines URL discovery tools (ParamSpider, Waybackurls, Gauplus, Hakrawler, Katana) wit… | 63 | 1863 | active |
| AnySearch AnySearch is a unified real-time search engine service for AI agents, distributed as an agent skill package and an MCP server. It provides … | 57 | 1860 | active |
| mwmbl/mwmbl Mwmbl is an open source, non-profit web search engine with no ads or tracking, where the community determines rankings and runs distributed… | 75 | 1858 | active |
| ZianTT/BHYG BHYG is a script/tool for automatically grabbing tickets for Bilibili World (BW) events via Bilibili's member purchase (会员购) platform. It a… | 67 | 1858 | active |
| NeteaseCloudMusicApiEnhanced/api-enhanced A Node.js API service providing comprehensive access to NetEase Cloud Music (网易云音乐) endpoints, a half-refactored and enhanced fork of the p… | 81 | 1855 | active |
| GuDaStudio/GrokSearch GrokSearch is an MCP server built on FastMCP that gives Claude Code and other LLM clients real-time web access via a dual-engine architectu… | 45 | 1852 | active |
| MDC-ng A self-hosted media metadata scraper and organizer for adult video libraries, written in Rust with a Next.js web UI. It scrapes metadata fr… | 77 | 1848 | active |
| LoseNine/ruyipage RuyiPage is a Python browser automation framework built on Firefox and the WebDriver BiDi protocol, shipping with an anti-detection Firefox… | 83 | 1844 | active |
| Aas-ee/open-webSearch Open-WebSearch is a TypeScript tool providing an MCP server, CLI, and local daemon for multi-engine web search and content retrieval withou… | 84 | 1842 | active |
| enetx/surf Surf is an advanced HTTP client library for Go with fluent, chainable API design. It supports browser impersonation (Chrome/Firefox), JA3/J… | 81 | 1840 | active |
| abinthomasonline/repo2txt A browser-based tool that converts GitHub, GitLab, Azure DevOps repositories, local folders, or ZIP files into a single formatted text file… | 63 | 1837 | active |
| microlinkhq/browserless A Node.js library that wraps Puppeteer to provide a production-ready headless Chrome/Chromium driver with built-in screenshot, PDF generati… | 92 | 1836 | active |
| tryolabs/requestium Requestium is a Python library that merges Requests, Selenium, and Parsel into a single integrated tool for web automation. It lets scripts… | 77 | 1833 | active |
| 78778443/QingScan QingScan is a self-hosted, open-source security operations platform that unifies vulnerability scanning, code auditing, asset inventory, an… | 62 | 1831 | active |
| lds133/weather_landscape A Python application that renders weather forecasts as a stylized landscape image instead of numeric dashboards, encoding time, temperature… | 53 | 1828 | active |
| ReScienceLab/opc-skills A collection of open-source Agent Skills (folders of instructions, scripts, and resources) designed for solopreneurs, indie hackers, and on… | 70 | 1825 | active |
| 1N3/BlackWidow BlackWidow is a Python-based web application spider that crawls a target site to collect URLs, dynamic parameters, subdomains, email addres… | 54 | 1824 | active |
| petronny/gfwlist2pac A tool that automatically converts the gfwlist proxy rules into a PAC (Proxy Auto-Config) file every day. The generated gfwlist.pac is serv… | 75 | 1819 | active |
| josh0xA/darkdump Darkdump is an open-source OSINT tool for querying multiple dark web search engines and scraping onion site results for emails, metadata, k… | 64 | 1798 | active |
| emacs-elfeed/elfeed Elfeed is an extensible web feeds client for Emacs supporting Atom, RSS, and JSON Feed formats. It provides a search-based UI inspired by n… | 67 | 1794 | active |
| deweizhu/bookget bookget is a Go-based command-line tool for downloading digitized ancient books and rare texts from 50+ digital libraries. It ships prebuil… | 57 | 1791 | active |
| kost/dvcs-ripper dvcs-ripper is a set of Perl command-line tools that download (rip) web-accessible version control repositories such as GIT, SVN, Mercurial… | 32 | 1789 | stable |
| LearnPrompt/ai-news-radar AI News Radar is an automated 24-hour AI/tech news aggregator that fetches sources, deduplicates stories, and scores headlines with three p… | 78 | 1787 | active |
| Masterminds/html5-php A standards-compliant HTML5 parser and serializer written entirely in PHP. It parses HTML5 documents and fragments into standard PHP DOM ob… | 81 | 1783 | stable |
| adsbypasser/adsbypasser AdsBypasser is a lightweight userscript that automatically skips countdown ads, continue/redirect pages, and prevents ad pop-up windows acr… | 99 | 1775 | active |
| bulianglin/psub psub is a proxy subscription conversion tool deployed on Cloudflare Workers that acts as a reverse proxy for subscription conversion backen… | 10 | 1769 | active |
| cambecc/earth Earth is a JavaScript web application that visualizes global weather, wind, ocean currents, and related atmospheric conditions on an animat… | 32 | 6613 | maintenance |
| website-scraper A Node.js library that downloads entire websites to a local directory, including HTML, CSS, images, and JavaScript assets. It parses HTTP r… | 68 | 1757 | active |
| oxylabs/google-play-scraper A free Python-based Google Play Store scraper that collects public app, movie, and book data via search queries. It is a companion tool to … | 64 | 1756 | active |
| DialmasterOrg/Youtarr Youtarr is a self-hosted web application that automatically downloads YouTube channel and playlist content, organizes it with metadata for … | 94 | 1753 | active |
| lit26/finvizfinance A Python library that scrapes and downloads financial data from the FinViz website, returning stock fundamentals, technicals, charts, news,… | 85 | 1753 | active |
| cxOrz/chaoxing-signin A Node.js/TypeScript tool that automates sign-in for the Chaoxing (Superstar Learning) online course platform, supporting normal, photo, ge… | 10 | 1751 | active |
| hzm0321/real-time-fund A Next.js web application for real-time mutual fund valuation and top-holdings stock tracking, primarily for Chinese funds, with portfolio,… | 79 | 1746 | active |
| Pageres A Node.js command-line tool that captures screenshots of websites at multiple resolutions using headless Chrome (Puppeteer). It is useful f… | 38 | 1746 | active |
| IvanGlinkin/Fast-Google-Dorks-Scan A shell-based OSINT tool that automates Google dork searches against a target website to uncover admin panels, exposed file types, and path… | 43 | 1745 | active |
| NotebookLM Book Importer A Python CLI tool that automatically downloads books from Z-Library and uploads them to Google NotebookLM in one command. It uses Playwrigh… | 41 | 1739 | active |
| egoist/sitefetch A Node.js CLI tool that crawls an entire website and saves its pages as a single text file, using Mozilla Readability to extract clean cont… | 21 | 1739 | active |