Open-source web-scraping software
2000 open-source libraries, tools and applications tagged web-scraping, ranked by how closely web-scraping is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
2000 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| hect0x7/JMComic-APK A GitHub Actions-based automation that periodically checks for updates to the JM Comic (18comic) Android APK and publishes new versions as … | 93 | 6589 | active |
| HapeLee/legado-with-MD3 Legado with MD3 is a free, open-source Android e-book and content aggregation reader, rebuilt from the Legado (阅读 3.0) project with a Mater… | 81 | 6506 | active |
| CharlesPikachu/musicdl A lightweight music downloader written in pure Python that supports dozens of music and audiobook platforms including NetEase Cloud Music, … | 72 | 6484 | active |
| gosom/google-maps-scraper An open-source Go tool that scrapes Google Maps to extract business data such as names, addresses, phone numbers, websites, ratings, review… | 95 | 6453 | active |
| s0md3v/Arjun Arjun is a Python command-line tool that discovers hidden HTTP query parameters for URL endpoints using a large dictionary of over 25,000 p… | 23 | 6402 | active |
| chyroc/WechatSogou A Python library providing a scraping API for WeChat official accounts based on Sogou WeChat search. It lets you look up account info and r… | 51 | 6398 | active |
| wechatsync/Wechatsync Wechatsync is an open-source Chrome extension that syncs articles from WeChat Official Accounts (or any webpage) to 29+ content platforms l… | 57 | 6380 | active |
| HakuNeko HakuNeko is a cross-platform desktop application (built on Electron) for downloading manga, anime, and novels from over 1200 websites via p… | 52 | 6372 | active |
| joeseesun/qiaomu-anything-to-notebooklm A Claude Code Skill that ingests content from 15+ sources (WeChat articles, web pages, YouTube, PDFs, EPUB, Office docs, audio) and uploads… | 69 | 6347 | active |
| nickscamara/open-deep-research An open-source clone of OpenAI's Deep Research, built as a Next.js web application. It uses Firecrawl's search and extract APIs to gather l… | 17 | 6291 | active |
| Nokogiri Nokogiri is a Ruby library for parsing, querying, modifying, and generating XML and HTML documents, built on native parsers like libxml2, l… | 92 | 6283 | stable |
| Bright Data MCP The official Bright Data CLI (npm package @brightdata/cli) providing terminal access to Bright Data's web scraping, search, and structured … | 81 | 6261 | active |
| Python3WebSpider/ProxyPool A self-hosted proxy pool service that periodically scrapes free proxy sites, stores and scores them in Redis, tests their availability, and… | 60 | 6252 | active |
| hanydd/BilibiliSponsorBlock A browser extension that automatically skips sponsored segments, intros/outros, and like-subscribe reminders in Bilibili videos, ported fro… | 94 | 6243 | active |
| bb-browser bb-browser is a TypeScript CLI and MCP server that lets AI agents control your real Chrome browser using your existing login state, exposin… | 64 | 6241 | active |
| truelockmc/streambert Streambert is a cross-platform Electron desktop application for streaming and downloading movies, TV series, and anime. It sources video st… | 75 | 6152 | active |
| BrowserAct BrowserAct Skills is a Python-based browser automation CLI designed for AI agents, providing real-browser control with anti-bot evasion (st… | 57 | 6118 | active |
| langren1353/GM_script A collection of Tampermonkey/Greasemonkey userscripts, most notably AC-baidu, which removes redirects and ads from Baidu, Sogou, Google, Bi… | 74 | 6058 | active |
| MontFerret/ferret Ferret is a declarative, expression-oriented query language (FQL) with an embeddable Go runtime for querying, transforming, and automating … | 79 | 6013 | active |
| microsoft/Webwright Webwright is a Python framework from Microsoft that turns coding LLMs into state-of-the-art browser agents by giving them a terminal to lau… | 54 | 6011 | active |
| Recon-ng Recon-ng is a full-featured, modular reconnaissance framework for conducting web-based open source intelligence (OSINT) gathering. It offer… | 76 | 6003 | active |
| AhmadIbrahiim/Website-downloader A Node.js web application that downloads the complete source code of any website, including all assets like JavaScripts, stylesheets, and i… | 72 | 5980 | active |
| asciimoo/hister Hister is a self-hosted, privacy-focused personal search engine that indexes the full contents of visited web pages and local files. It off… | 82 | 5936 | active |
| RedSiege/EyeWitness EyeWitness is a Python CLI tool that takes screenshots of websites using headless Chromium, captures server header information, and identif… | 46 | 5858 | active |
| qiye45/wechatVideoDownload A desktop GUI tool for downloading WeChat Channels (视频号) videos, live streams, live replays, and images. It automatically monitors WeChat t… | 80 | 5828 | active |
| 60s API A collection of free, open-source REST APIs delivering daily news digests, Chinese social media trending lists (Weibo, Bilibili, Douyin, Zh… | 89 | 5779 | active |
| vantage-sh/ec2instances.info EC2Instances.info is an open-source web application for comparing Amazon EC2, RDS, and ElastiCache instance types, specs, and pricing acros… | 76 | 5766 | active |
| FxEmbed FxEmbed is a Cloudflare Worker service that powers FxTwitter, FixupX, and FxBluesky, rewriting X/Twitter and Bluesky links into rich embeds… | 77 | 5751 | active |
| omkarcloud/botasaurus Botasaurus is an all-in-one Python web scraping framework with built-in anti-detection, caching, parallelization, and proxy support. It let… | 69 | 5721 | active |
| TechXueXi TechXueXi is an open-source Python automation tool that automatically completes daily study tasks and quizzes on China's Xuexi Qiangguo (学习… | 52 | 5668 | active |
| Pixiv Batch Downloader A browser extension (Chrome, Edge, Firefox) for batch downloading illustrations, manga, Ugoira animations, and novels from Pixiv. It offers… | 97 | 5664 | active |
| executeautomation/mcp-playwright A Model Context Protocol (MCP) server that exposes Playwright browser automation to LLM clients like Claude Desktop, Cline, and Cursor IDE.… | 44 | 5664 | active |
| rmax/scrapy-redis Redis-based components for Scrapy that enable distributed crawling and scraping by sharing a Redis queue across multiple spider instances. … | 66 | 5644 | active |
| Potree Potree is a free open-source WebGL-based renderer for viewing large point clouds directly in web browsers. It uses octree-based level-of-de… | 47 | 5618 | active |
| GitHub Contributions Chart A web application that generates a shareable image of a user's entire GitHub contribution history, with multiple color themes. It works aro… | 32 | 5609 | active |
| Neet-Nestor/Telegram-Media-Downloader A browser userscript (installable via Tampermonkey, Violentmonkey, etc.) that adds media download buttons to the Telegram webapp. It enable… | 61 | 5555 | active |
| tangyoha/telegram_media_downloader A cross-platform Python application that downloads media files (video, audio, photos, documents) from Telegram chats, channels, and private… | 50 | 5555 | active |
| AngleSharp AngleSharp is a .NET library that parses HTML5, SVG, MathML, XML, and CSS into a fully standards-compliant W3C DOM. It supports querySelect… | 98 | 5541 | active |
| 88lin/video_vip A Tampermonkey/Greasemonkey userscript that integrates multiple third-party parsing interfaces to bypass VIP membership restrictions on Chi… | 64 | 5531 | active |
| tebelorg/RPA-Python A Python package for robotic process automation (RPA) that wraps TagUI to automate web pages, desktop apps, and visual elements via a simpl… | 62 | 5504 | active |
| jef/streetmerchant streetmerchant is a Node.js application that continuously checks retail websites for product stock availability, with optional add-to-cart … | 60 | 5402 | active |
| spatie/browsershot A PHP library that converts web pages or raw HTML into images, PDFs, or rendered HTML strings using Puppeteer and headless Chrome. It suppo… | 85 | 5246 | active |
| Obsidian Obsidian Web Clipper is the official browser extension for Obsidian that lets users highlight web pages and capture content as durable Mark… | 85 | 5230 | active |
| Yuukiy/JavSP JavSP is a Python command-line tool that scrapes adult video (JAV) metadata from multiple websites, aggregates the data, and generates NFO … | 23 | 5173 | active |
| hakluke/hakrawler Hakrawler is a fast command-line web crawler written in Go, built on the Gocolly library, that discovers URLs and JavaScript file locations… | 62 | 5146 | active |
| ShadowHackrs/gmail-account-creator A Python-based automation tool for bulk creation of Gmail accounts, featuring anti-detection techniques like human-like typing simulation a… | 59 | 5137 | active |
| lc/gau gau (getallurls) is a Go CLI tool that fetches known URLs for a given domain from AlienVault's Open Threat Exchange, the Wayback Machine, C… | 67 | 5127 | active |
| scinfu/SwiftSoup SwiftSoup is a pure Swift HTML parser library that conforms to the WHATWG HTML5 specification and offers DOM traversal, CSS selectors, and … | 95 | 5126 | stable |
| exa-labs/exa-mcp-server An open-source MCP (Model Context Protocol) server that connects AI assistants and agents to Exa's web search, code search, content fetchin… | 66 | 5112 | active |
| binbyu/Reader A lightweight, free, open-source Windows (win32) reader application for txt and epub files, with support for reading online novels via conf… | 65 | 5109 | active |
| DedSecInside/TorBot TorBot is a Python CLI tool for OSINT on the dark web, crawling .onion sites over the Tor network and building link trees. It can save craw… | 95 | 5045 | active |
| FellouAI/eko Eko is a production-ready JavaScript framework for building reliable AI agents and agentic workflows from natural language, running in both… | 57 | 4961 | active |
| davidarroyo1234/InstagramUnfollowers A browser-based script that scans your Instagram account to identify users who don't follow you back, letting you selectively unfollow them… | 76 | 4935 | active |
| jaypyles/Scraperr Scraperr is a self-hosted web scraping application with a web UI that lets users scrape websites without writing code, using XPath-based ex… | 10 | 4910 | active |
| Patchright Patchright is a patched, undetected fork of the Playwright browser automation framework that evades bot-detection systems like Cloudflare. … | 95 | 4904 | active |
| MechanicalSoup/MechanicalSoup A Python library for automating interaction with websites, built on Requests and BeautifulSoup. It handles cookies, redirects, link followi… | 62 | 4894 | active |
| SocialFish SocialFish is a Python-based phishing toolkit that clones modern login pages using Playwright browser automation and captures credentials, … | 70 | 4892 | active |
| rachelos/we-mp-rss A self-hosted WeChat official account (公众号) subscription assistant that scrapes articles, generates RSS feeds, and converts content to Mark… | 83 | 4868 | active |
| ultrafunkamsterdam/nodriver Nodriver is a fully asynchronous Python browser automation and web scraping library, and the official successor to Undetected-Chromedriver.… | 59 | 4799 | active |
| Keiyoushi Extensions A community-maintained repository of extensions (APKs) for Mihon and its forks, providing manga source plugins. The source code for the ext… | 72 | 4797 | active |
| fb55/htmlparser2 htmlparser2 is a fast, forgiving HTML and XML parser for JavaScript and TypeScript, offering a low-allocation callback interface as well as… | 84 | 4792 | stable |
| Integuru-AI/Integuru Integuru is an AI agent that reverse-engineers platforms' internal APIs by analyzing browser network requests (HAR files) and building depe… | 59 | 4781 | active |
| HTTrack Website Copier HTTrack is a free offline browser utility that recursively downloads websites to a local directory, rewriting links so the mirrored copy ca… | 99 | 4780 | stable |
| l0o0/translators_CN A community-maintained collection of Zotero translators for Chinese academic and general websites, enabling Zotero to scrape citation metad… | 76 | 4776 | active |
| bjesus/pipet Pipet is a command-line web scraper written in Go that extracts data from online assets using HTML parsing, JSON parsing, and client-side J… | 14 | 4768 | active |
| 201206030/novel-plus novel-plus is a full-featured novel/fiction CMS built on Spring Boot, comprising a reader-facing portal, author backend, admin dashboard, a… | 100 | 4756 | active |
| Twikit Twikit is a free Python library that wraps Twitter's internal API, allowing posting, searching, and scraping tweets without an official API… | 57 | 4720 | active |
| danburzo/percollate Percollate is a Node.js command-line tool that converts web pages into readable, well-formatted PDF, EPUB, HTML, or Markdown documents. It … | 48 | 4686 | active |
| lecepin/WeChatVideoDownloader A convenient desktop GUI application for downloading videos from WeChat Channels (WeChat Video Accounts). It intercepts and captures video … | 10 | 4669 | active |
| dataabc/weibo-crawler A Python crawler for Sina Weibo that scrapes user profiles and posts, exporting data to CSV, JSON, MySQL, MongoDB, or SQLite, and optionall… | 70 | 4663 | active |
| joeyism/linkedin_scraper A Python library that scrapes LinkedIn for user, company, and job data using Playwright with an async API. It provides Pydantic data models… | 75 | 4603 | active |
| linsomniac/spotify_to_ytmusic A set of Python scripts and a GUI for copying liked songs and playlists from Spotify to YouTube Music. It uses the Spotify backup data and … | 60 | 4584 | active |
| syhyz1990/baiduyun A free open-source Tampermonkey userscript that extracts real direct download links from Chinese cloud storage services (Baidu Netdisk, Ali… | 32 | 4571 | active |
| limbopro/Adblock4limbo A userscript-based ad-blocking project that removes popups, banners, and video ads on specific streaming, comic, novel, and adult sites via… | 77 | 4542 | active |
| Anakin-Inc/anakin AnakinScraper OSS is a self-hosted web scraping API written in Go that turns any website into LLM-ready markdown or structured JSON via a s… | 68 | 4535 | active |
| sensepost/gowitness gowitness is a Go-based command-line utility that uses Chrome Headless to take screenshots of websites, supporting scans of URL lists, CIDR… | 85 | 4522 | active |
| 6dylan6/jdpro A collection of JavaScript automation scripts for the Qinglong panel, primarily for running scheduled JD (Jingdong) sign-in and reward task… | 73 | 4517 | active |
| speedyapply/JobSpy JobSpy is a Python library that scrapes job postings from popular job boards like LinkedIn, Indeed, Glassdoor, Google, and ZipRecruiter con… | 97 | 4514 | active |
| metatube-community/jellyfin-plugin-metatube A metadata provider plugin for Jellyfin and Emby media servers that fetches movie and actor metadata from various internet providers via th… | 55 | 4498 | active |
| GerbenJavado/LinkFinder LinkFinder is a Python CLI script that discovers endpoints and their parameters in JavaScript files using jsbeautifier and regular expressi… | 32 | 4464 | stable |
| sparklemotion/mechanize Mechanize is a Ruby library for automating interaction with websites. It handles cookies, redirects, link following, and form submission wh… | 82 | 4443 | stable |
| VonChange/utao Utao TV (油桃TV, now upgraded to 土拨鼠浏览器) is a third-party browser designed for Android TV boxes and smart TVs that lets users watch live CCTV… | 64 | 4418 | active |
| RxJS RxJS is a JavaScript library for composing asynchronous and event‑based programs using observable sequences and functional operators. It pr… | 10 | 19326 | maintenance |
| hafrey1/LunaTV-config A configuration repository and Cloudflare Workers-based CORS proxy for MoonTV/LunaTV video source APIs, with daily automated API health che… | 64 | 4373 | active |
| QasimWani/LeetHub LeetHub is a browser extension that automatically pushes your LeetCode solutions to GitHub whenever you pass all tests on a problem. It sup… | 68 | 4353 | active |
| arxhr007/Aliens_eye Aliens Eye is an AI-powered OSINT CLI tool that scans 840+ social media and web platforms to find accounts associated with a given username… | 99 | 4344 | active |
| hoothin/UserScripts A collection of Greasemonkey/Tampermonkey userscripts by hoothin, including Pagetual (auto-pager infinite scrolling), Picviewer CE+ (online… | 76 | 4329 | active |
| anasty17/mirror-leech-telegram-bot A Python Telegram bot that mirrors or leeches files from direct links, torrents, NZB/Usenet, Google Drive, rclone clouds, and yt-dlp/JDownl… | 76 | 4318 | active |
| Lucksi/Mr.Holmes Mr.Holmes is a Python-based OSINT (open-source intelligence) CLI tool that gathers information about usernames, domains, phone numbers, and… | 50 | 4314 | active |
| kanasimi/work_crawler A multi-language downloader application that batch-downloads web novels (converting them to EPUB) and comics from a large list of Chinese, … | 67 | 4303 | active |
| wasi-master/13ft A self-hosted web service that bypasses paywalls on news sites by fetching pages as GoogleBot, a replacement for the defunct 12ft.io. It se… | 92 | 4301 | active |
| morpheus65535/bazarr Bazarr is a self-hosted companion application to Sonarr and Radarr that automatically manages and downloads subtitles for your TV series an… | 93 | 4300 | active |
| megadose/toutatis Toutatis is a Python CLI tool that extracts public information from Instagram accounts, such as emails, phone numbers, follower counts, and… | 32 | 4298 | active |
| UltimaHoarder/UltimaScraper A Python-based scraper that downloads all media (photos, videos) from OnlyFans accounts using the user's own session authentication. It sto… | 23 | 4278 | active |
| Manavarya09/design-extract designlang is an open-source CLI (npx designlang <url>) that uses a Playwright-driven headless browser to reverse-engineer any website's de… | 81 | 4275 | active |
| jasonxtn/Argus Argus is a Python-based all-in-one information gathering and reconnaissance toolkit with an interactive console and modular architecture. I… | 44 | 4264 | active |
| praw-dev/praw PRAW is a Python package that wraps Reddit's API, providing simple, rule-compliant access to Reddit data and actions. It handles OAuth auth… | 95 | 4259 | stable |
| Guyungy/damaihelper DamaiHelper is a multi-platform ticket-grabbing automation assistant (Damai, Taopiaopiao, Binwandao) built as a Python backend with an Ant … | 71 | 4257 | active |
| CodeceptJS CodeceptJS is a Node.js end-to-end testing framework with a BDD-style syntax where tests are written as user-perspective scenarios using an… | 99 | 4244 | active |
| vogler/free-games-claimer A Node.js automation tool that periodically claims free games and DLCs on the Epic Games Store, Amazon Prime Gaming, and GOG, plus free Unr… | 71 | 4238 | active |