Open-source web-scraping software
2000 open-source libraries, tools and applications tagged web-scraping, ranked by how closely web-scraping is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
2000 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| m4ll0k/SecretFinder SecretFinder is a Python CLI script based on LinkFinder that discovers sensitive data like API keys, access tokens, and JWTs in JavaScript … | 32 | 2534 | active |
| BetterBahn/betterbahn BetterBahn is an open-source Next.js web app for finding train journeys in Germany, with a focus on split-ticketing to help users save mone… | 62 | 2532 | active |
| guyueyingmu/avbook A self-hosted PHP/Laravel web application that manages a Japanese adult video (JAV) library, backed by crawlers for sites like avmoo, javbu… | 23 | 10079 | maintenance |
| fhamborg/news-please news-please is an open-source Python news crawler and information extractor that pulls structured article data (headline, lead, main text, … | 63 | 2493 | active |
| tid-kijyun/Kanna Kanna is a Swift XML/HTML parser library inspired by Ruby's Nokogiri, built on libxml2. It supports XPath 1.0 and CSS3 selector queries acr… | 66 | 2486 | active |
| Bistutu/GoMusic GoMusic is a web application that migrates playlists from Chinese music services (NetEase Cloud, Qishui, QQ Music) to Apple Music, YouTube … | 56 | 2476 | active |
| hisxo/gitGraber gitGraber is a Python3 command-line tool that monitors GitHub search results in real time to find leaked sensitive data such as API keys an… | 62 | 2452 | active |
| kurtmckee/feedparser feedparser is a Python library that parses RSS, Atom, RDF, CDF, and JSON feeds into a uniform data structure. It is the de facto standard u… | 86 | 2444 | stable |
| dimdenGD/OldTweetDeck A browser extension that restores the classic (old) TweetDeck interface on X/Twitter. It works on Chromium-based browsers and Firefox by in… | 84 | 2442 | active |
| ccloli/E-Hentai-Downloader A userscript that downloads E-Hentai/ExHentai galleries as zip files directly from the browser. It runs via userscript managers like Tamper… | 56 | 2442 | active |
| coursera-dl/coursera-dl A Python command-line script for batch downloading Coursera.org lecture videos and resources with proper naming. It supports resuming downl… | 32 | 9655 | maintenance |
| lmnr-ai/index Index is an open-source browser agent that autonomously performs complex tasks on the web by turning any website into an accessible API. It… | 10 | 2438 | active |
| iliane5/meridian Meridian is an open-source AI-powered news intelligence system that scrapes hundreds of RSS sources, clusters and analyzes articles with LL… | 28 | 2437 | active |
| 3441293738/creatorhub CreatorHub is a self-hosted web panel built with Python and FastAPI for managing, monitoring, scraping, downloading, and publishing content… | 81 | 2422 | active |
| rust-scraper/scraper A Rust library for parsing HTML documents and querying them with CSS selectors, built on Servo's html5ever and selectors crates for browser… | 86 | 2421 | active |
| jarvis2f/telegram-files A self-hosted web application for downloading files from Telegram channels and groups continuously and unattended. It supports multiple Tel… | 86 | 2415 | active |
| tavily-ai/tavily-mcp A Model Context Protocol (MCP) server that exposes Tavily's web search, page extraction, site mapping, and crawling tools to AI assistants … | 65 | 2415 | active |
| OpenBullet OpenBullet 2 is a cross-platform automation suite built on .NET for performing HTTP requests against target web applications and processing… | 91 | 2411 | active |
| Apify Agent Skills A collection of production-grade agent skills from Apify that give AI coding agents (Claude Code, Cursor, Windsurf, Codex, Gemini CLI) expe… | 60 | 2408 | active |
| scrapinghub/portia Portia is a visual web scraping tool from Scrapinghub built on Scrapy that lets users annotate web pages in a browser to define data extrac… | 10 | 9501 | maintenance |
| kevinho/clawfeed ClawFeed is an AI-powered news digest application that curates sources like Twitter, RSS, HackerNews, Reddit, and GitHub Trending into stru… | 59 | 2398 | active |
| Xiangyu-CAS/xiaohongshu-ops-skill A skill for the OpenClaw agent that turns it into a Xiaohongshu (RedNote) operations assistant, using browser automation (CDP) to analyze f… | 45 | 2385 | active |
| vitiko98/qobuz-dl A Python command-line tool for searching, exploring, and downloading lossless and Hi-Res music (FLAC/MP3) from Qobuz. It supports interacti… | 34 | 2385 | active |
| 0xMassi/webclaw webclaw is a Rust-based web extraction toolkit that turns any URL into clean, LLM-ready markdown, JSON, or token-optimized text, including … | 77 | 2379 | active |
| gawel/pyquery pyquery is a Python library that provides a jQuery-like API for querying and manipulating XML and HTML documents, built on top of lxml for … | 77 | 2376 | stable |
| amaancoderx/npxskillui SkillUI is a Node.js CLI that crawls websites, git repos, or local codebases and extracts their complete design system (colors, typography,… | 48 | 2375 | active |
| lijiejie/BBScan BBScan is a fast, lightweight, high-concurrency web vulnerability scanner written in Python. It helps penetration testers quickly identify … | 23 | 2374 | active |
| StractOrg/stract Stract is an open-source web search engine written in Rust with its own independent crawler and index, built on the Tantivy inverted index … | 10 | 2371 | active |
| gxr404/yuque-dl yuque-dl is a Node.js CLI tool that downloads Yuque (语雀) knowledge bases and documents as local Markdown files. It supports batch downloads… | 72 | 2369 | active |
| degoog-org/degoog Degoog is a self-hosted search aggregator that queries multiple search engines in parallel from your own server and merges results into a s… | 82 | 2367 | active |
| EtherDream/jsproxy An online web proxy that uses browser Service Workers to intercept and rewrite requests client-side, so the nginx-based server only forward… | 32 | 9324 | maintenance |
| ying-ck/fanqienovel-downloader A Python tool that downloads novels from Fanqie Novel (fanqienovel.com) by URL or book ID, with search, batch download, update, and backup … | 83 | 2364 | active |
| VermiIIi0n/fuckZHS A Python 3 automation script that automatically completes Zhihuishu (智慧树) online course videos, including auto-answering pop-up quiz questi… | 41 | 2354 | active |
| miraclx/freyr-js Freyr is a Node.js CLI tool that downloads songs from music streaming services like Spotify, Apple Music, and Deezer. It extracts track met… | 66 | 2350 | active |
| lcomplete/huntly Huntly is a self-hosted, AI-powered personal information hub that captures, archives, and organizes web content, RSS feeds, tweets, and hig… | 81 | 2348 | active |
| FriendsOfPHP/Goutte Goutte is a PHP screen scraping and web crawling library providing a simple API to crawl websites and extract data from HTML/XML responses.… | 10 | 9180 | maintenance |
| Taiga Taiga is an open-source Windows desktop application that automatically detects anime videos you watch on your computer and syncs your progr… | 66 | 2335 | active |
| dataabc/weibo-search A Python/Scrapy-based crawler that continuously fetches Weibo keyword and hashtag search results, including full post metadata, images, and… | 67 | 2328 | active |
| easychen/checkchan-dist Check酱 (CheckChan) is a web page content change monitoring tool consisting of a Chrome/Edge browser extension and a self-hostable cloud com… | 32 | 2323 | active |
| saifyxpro/HeadlessX HeadlessX is a self-hosted browser automation and web scraping platform powered by Camoufox (a C++-patched Firefox) to bypass anti-bot syst… | 72 | 2321 | active |
| SteamTradingSiteTracker A Steam skin trading market tracker that continuously monitors sell-to-cash (挂刀) ratios across BUFF, IGXE, C5, UUYP, and ECO platforms for … | 32 | 2319 | active |
| jaywcjlove/github-rank A GitHub user and repository ranking site that publishes global and China leaderboards of GitHub users by followers and repositories by sta… | 99 | 2316 | active |
| sjdirect/abot Abot is an open source C# web crawler framework built for speed and flexibility, handling multithreading, HTTP requests, scheduling, and li… | 70 | 2309 | active |
| NAStool/nas-tools A self-hosted NAS media library management tool that automates organizing, scraping, and syncing media content for media servers like Plex,… | 10 | 9018 | maintenance |
| ajayyy/DeArrow DeArrow is an open-source browser extension that crowdsources better titles and thumbnails for YouTube videos, replacing sensationalized cl… | 94 | 2299 | active |
| anaskhan96/soup soup is a small Go library for web scraping with an API modeled after Python's BeautifulSoup. It fetches HTML over HTTP and builds a DOM th… | 95 | 2297 | active |
| Future-Scholars/paperlib Paperlib is an open-source, cross-platform desktop application for managing academic papers, built with TypeScript and Electron. It scrapes… | 54 | 2297 | active |
| AaronL725/grok-register A Python toolkit that automates bulk registration of Grok accounts using real Chromium browser automation, with GUI, CLI, and WebUI interfa… | 57 | 2286 | active |
| goclone-dev/goclone Goclone is a Go CLI utility that downloads entire websites to a local directory, preserving relative link structure so the mirrored site ca… | 57 | 2277 | active |
| mampfes/hacs_waste_collection_schedule A Home Assistant custom integration that fetches waste collection schedules from many service providers, ICS/iCal files, or user-defined da… | 98 | 2271 | active |
| Johnserf-Seed/TikTokDownload A Python CLI tool for batch-downloading Douyin (Chinese TikTok) content without watermarks, including user profile posts, likes, favorites,… | 23 | 8876 | maintenance |
| lanyeeee/bilibili-video-downloader A cross-platform GUI desktop application built with Tauri for downloading videos, audio, subtitles, danmaku, and covers from Bilibili. It s… | 70 | 2266 | active |
| MinhasKamal/DownGit DownGit is a web tool that creates direct download links for any public GitHub directory or file, packaging them as a zip archive. It is ho… | 50 | 2266 | active |
| Archon An MCP server that combines Crawl4AI web crawling with RAG capabilities backed by a Supabase vector database, exposing tools for AI agents … | 31 | 2266 | active |
| vasu-devs/JustHireMe JustHireMe is a local-first desktop workbench (Tauri frontend, Python backend) that scrapes job postings, ranks role fit against your profi… | 75 | 2256 | stable |
| Ocyss/boss-helper A browser extension (also available as a userscript) that enhances the Boss Zhipin job platform by removing ads, improving the UI, enabling… | 85 | 2249 | active |
| hhursev/recipe-scrapers A Python library for extracting structured recipe data (title, ingredients, instructions, cooking times, images, nutrients) from cooking we… | 96 | 2247 | active |
| wabarc/wayback Wayback is an open-source web archiving tool written in Go that captures and preserves web pages via services like Internet Archive, archiv… | 83 | 2241 | active |
| hangone/WeBan WeBan is a Python-based automation tool that automatically completes courses and exams on the Weiban (安全微伴) university safety education pla… | 83 | 2241 | active |
| iuroc/bilidown Bilidown is a Bilibili video parsing and downloading desktop application supporting 8K video, Hi-Res audio, Dolby Vision, batch parsing, QR… | 78 | 2237 | active |
| dou-jiang/codex-console A Python-based integrated console for automating OpenAI/Codex account registration, login, token retrieval, subscription management, and up… | 67 | 2234 | active |
| dmzz-yyhyy/LightNovelReader LightNovelReader is an open-source Android light novel reader app built with Kotlin and Jetpack Compose, featuring a lightweight footprint … | 97 | 2223 | active |
| AnotiaWang/deep-research-web-ui A web UI for the dzhng/deep-research project that performs iterative, AI-driven research by combining search engines, web scraping, and LLM… | 87 | 2223 | active |
| Hubs Hubs is an open-source, browser-based multi-user 3D virtual world and social VR platform built with A-Frame, Three.js, and WebXR/WebRTC. It… | 76 | 2218 | active |
| vibheksoni/stealth-browser-mcp A Python MCP server that exposes stealth browser automation (via nodriver and Chrome DevTools Protocol) to AI agents, letting them navigate… | 61 | 2213 | active |
| ReaJason/xhs A Python SDK that wraps requests to the Xiaohongshu (Little Red Book) web platform for extracting data. It provides a programmatic client f… | 40 | 2213 | active |
| MarginaliaSearch/MarginaliaSearch Marginalia Search is an independent, open-source internet search engine that indexes text-oriented, non-commercial, small and old websites.… | 66 | 2205 | active |
| AAndyProgram/SCrawler SCrawler is a Windows GUI application that downloads photos and videos from user profiles across many social media and content sites, inclu… | 93 | 2197 | active |
| Imangazaliev/DiDOM DiDOM is a fast and simple PHP library for parsing and manipulating HTML and XML documents. It supports loading from strings, files, or URL… | 48 | 2195 | active |
| opennaslab/kubespider Kubespider is a self-hosted download orchestration system that turns an idle Linux server into a NAS download center. It uses pluggable sou… | 54 | 2190 | active |
| Owez/yark Yark is a Python CLI tool for archiving YouTube channels, downloading videos and accumulating metadata over time with change reports. It in… | 62 | 2189 | active |
| bigintpro/csdn_downloader A Java (Spring Boot + Dubbo) based web service that downloads CSDN resources such as articles and paid/VIP documents without needing points… | 55 | 2179 | active |
| levigross/grequests GRequests is a Go library that wraps net/http with a convenient, Python Requests-style API. It provides helpers for all HTTP verbs, JSON/XM… | 10 | 2179 | active |
| watercrawl/WaterCrawl WaterCrawl is a self-hostable web application (Python/Django/Scrapy/Celery) that crawls websites and transforms web content into LLM-ready … | 77 | 2177 | active |
| NobyDa/Script A collection of JavaScript scripts and configuration files for iOS proxy tools such as Surge, Quantumult X, Loon, Stash, and Shadowrocket. … | 72 | 8448 | maintenance |
| POf-L/Fanqie-novel-Downloader A cross-platform desktop and mobile application built with Rust and Tauri v2 for searching, reading, and downloading novels from Fanqie Nov… | 81 | 2176 | active |
| ericchiang/pup pup is a command line tool for parsing and filtering HTML using CSS selectors, inspired by jq. It reads HTML from stdin, applies selector-b… | 23 | 8437 | maintenance |
| mgz0227/legado-Harmony Legado (开源阅读) for HarmonyOS is a free, open-source novel and ebook reader application. It supports custom book sources with user-defined sc… | 88 | 2170 | active |
| zhzyker/dismap Dismap is a Go-based asset discovery and identification tool that fingerprints web, TCP, UDP, and TLS services using a rule base of 4500+ w… | 23 | 2165 | active |
| drunkdream/weread-exporter A Python CLI tool that exports books from WeChat Read (微信读书) into epub, pdf, and mobi formats. It hooks into the web reader's Canvas render… | 61 | 2163 | active |
| Rongronggg9/RSS-to-Telegram-Bot A self-hosted Telegram bot that delivers RSS/Atom feed updates to Telegram chats with rich-text formatting and media support. It is multi-u… | 66 | 2160 | active |
| benvinegar/counterscale Counterscale is a self-hosted, privacy-friendly web analytics tracker and dashboard built on Cloudflare Workers and Workers Analytics Engin… | 78 | 2155 | active |
| cxfksword/jellyfin-plugin-metashark A Jellyfin metadata plugin that scrapes movie and anime metadata primarily from Douban, with TheMovieDb used to fill in missing episode dat… | 98 | 2154 | active |
| bit4woo/domain_hunter_pro Domain Hunter Pro is a Burp Suite plugin (Java jar) for automated domain and subdomain collection, web title fetching, and target managemen… | 67 | 2151 | active |
| philss/floki Floki is an Elixir HTML parser that lets you search document nodes using CSS selectors. It supports multiple parsing backends (mochiweb_htm… | 87 | 2148 | stable |
| up209d/ResourcesSaverExt A Chrome browser extension that downloads all resources of a website with one click while preserving the original folder structure. It inte… | 34 | 2148 | active |
| pablouser1/ProxiTok ProxiTok is an open-source alternative frontend for TikTok, inspired by Nitter, written in PHP. It proxies all requests to TikTok server-si… | 30 | 2142 | active |
| prajwalch/TorrentSearch TorrentSearch is an Android app that searches torrents across multiple providers simultaneously, with category filters, detailed results, b… | 83 | 2139 | active |
| walkingddd/TgtoDrive TgtoDrive is a self-hosted, Docker-deployed media automation platform that chains resource discovery (Telegram channel monitoring, search s… | 73 | 2138 | active |
| vvoovv/blosm Blosm is a Blender addon that imports real-world geodata—OpenStreetMap buildings and roads, Google 3D city tiles, and terrain—with global c… | 65 | 2128 | active |
| MaoTai_GUI A Windows GUI/console application for automated flash-sale purchasing (sniping) on JD, Taobao, and Damai, distributed as a ready-to-run EXE… | 71 | 2125 | active |
| yuanzl77/IPTV A Python tool that aggregates IPTV live streaming sources daily, validates them with HTTP checks and FFprobe quality probing, and generates… | 69 | 2122 | active |
| oxylabs/how-to-scrape-google-images A Python-based command-line tool that scrapes Google Images search results, including reverse image search based on a provided image URL. I… | 61 | 2117 | active |
| lexbor/lexbor Lexbor is a fast, standards-compliant HTML parser and DOM library written in pure C99, with additional modules for CSS, URL, encoding, and … | 80 | 2111 | active |
| AzizKpln/Moriarty-Project Moriarty Project is a web-based phone number investigation tool written in Python that gathers information about a given phone number. It a… | 65 | 2110 | active |
| oxylabs/how-to-scrape-google-flights A Python-based free scraper tool and tutorial for extracting flight data (prices, times, airlines) from Google Flights pages, either direct… | 55 | 2110 | active |
| feedjira/feedjira Feedjira is a Ruby library for parsing syndication feeds such as RSS and Atom. It supports extensible and custom parsers, letting users add… | 75 | 2100 | active |
| ipfs/public-gateway-checker A web application that displays a list of public IPFS gateways and checks whether each is online, including CORS, IPNS, origin isolation, a… | 95 | 2099 | active |
| hanc00l/nemo_go Nemo is an automated information-gathering platform for penetration testing that integrates common recon tools (Masscan, Nmap, Subfinder, H… | 82 | 2098 | active |
| AWeirdDev/flights fast-flights is a Python library that scrapes Google Flights by generating Base64-encoded Protobuf query strings, returning strongly-typed … | 83 | 2096 | active |