Open-source web-scraping software
2000 open-source libraries, tools and applications tagged web-scraping, ranked by how closely web-scraping is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
2000 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| debridmediamanager/debrid-media-manager Debrid Media Manager (DMM) is a free, open-source web app for curating an unlimited-size movie and TV show library on debrid services like … | 75 | 1456 | active |
| Roach Roach is a complete web scraping and crawling toolkit for PHP, heavily inspired by Python's Scrapy. It lets developers define spiders that … | 41 | 1456 | active |
| tychxn/jd-assistant A JD.com (Jingdong) purchase assistant written in Python that automates login via QR code, product stock/price queries, cart management, an… | 32 | 5262 | maintenance |
| Zendriver Zendriver is an async-first Python web scraping and browser automation framework built on the Chrome Devtools Protocol, forked from nodrive… | 88 | 1455 | active |
| scrapy-plugins/scrapy-playwright A Scrapy download handler that uses Playwright for Python to fetch pages, enabling scraping of JavaScript-rendered sites while keeping the … | 83 | 1452 | active |
| gpodder/gpodder gPodder is a free, open-source podcast client and media aggregator that lets users subscribe to, download, and manage podcast feeds. It has… | 66 | 1451 | stable |
| 6551Team/opentwitter-mcp A Python MCP server that exposes Twitter/X data (user profiles, tweet search, follower events, deleted tweets, KOL tracking) to AI assistan… | 59 | 1448 | active |
| nianzhibai/91 A self-hosted private video site written in Go that aggregates videos from multiple cloud drives (115, PikPak, 123pan, OneDrive, Google Dri… | 79 | 1447 | active |
| sw33tLie/bbscope bbscope is a Go CLI tool that fetches, stores, and manages bug bounty program scopes from HackerOne, Bugcrowd, Intigriti, YesWeHack, and Im… | 74 | 1446 | active |
| ShilongLee/Crawler A self-hostable crawler API server that exposes HTTP endpoints for scraping public data from Douyin, Kuaishou, Bilibili, Xiaohongshu, Weibo… | 58 | 1446 | active |
| drawrowfly/tiktok-scraper A TypeScript library and CLI tool that scrapes TikTok metadata from user, hashtag, trend, and music pages and downloads video posts without… | 23 | 5215 | maintenance |
| zhaoolee/garss Garss (嘎!RSS) is a self-hosted RSS aggregation and reading system that uses GitHub Actions to collect hundreds of RSS feeds and render them… | 80 | 1439 | active |
| rebrowser A collection of source-code patches for Puppeteer and Playwright that fix automation leaks and help avoid bot detection systems like Cloudf… | 31 | 1433 | active |
| facundoolano/app-store-scraper A Node.js library for scraping application data from the iTunes and Mac App Store. It provides methods to retrieve app details, search resu… | 44 | 1432 | active |
| blacklanternsecurity/MANSPIDER MANSPIDER is a Python CLI tool that crawls SMB shares across entire networks to find files by filename or content, with regex support and t… | 77 | 1428 | active |
| wd210010/only_for_happly A collection of Python automation scripts designed to run on the Qinglong (青龙) panel, covering daily check-ins for services like Baidu Tieb… | 71 | 1428 | active |
| stereobooster/react-snap A zero-configuration, framework-agnostic static prerendering tool for single-page applications. It uses Headless Chrome (via Puppeteer) to … | 58 | 5119 | maintenance |
| openwpm/OpenWPM OpenWPM is a web privacy measurement framework built on Firefox with Selenium automation, designed to collect data from thousands to millio… | 91 | 1420 | active |
| browserwing/browserwing BrowserWing is an open-source browser automation platform written in Go with a React dashboard that exposes browser control via MCP command… | 65 | 1419 | active |
| lorenzodifuccia/safaribooks A Python CLI tool that downloads books from O'Reilly Learning (Safari Books Online) and generates EPUB files from them. It requires a valid… | 47 | 5107 | maintenance |
| jocmp/capyreader Capy Reader is a free, open-source RSS feed reader and news aggregator for Android built with Kotlin and Jetpack Compose. It syncs with Fee… | 95 | 1416 | active |
| jez500/pricebuddy PriceBuddy is a self-hostable web application that tracks product prices and availability across online stores, keeping price history and n… | 88 | 1416 | active |
| danny0838/content-farm-terminator Content Farm Terminator is a cross-platform browser extension that identifies content farms by marking hyperlinks pointing to them and bloc… | 76 | 1415 | active |
| sec-edgar/sec-edgar A Python library and CLI for downloading company periodic reports, filings, and forms from the SEC's EDGAR database. It supports fetching f… | 44 | 1415 | active |
| superhedgy/AttackSurfaceMapper AttackSurfaceMapper is a Python CLI reconnaissance tool that expands a target's attack surface using OSINT and active techniques like subdo… | 32 | 1412 | active |
| tavily-ai/tavily-python The official Python SDK for the Tavily API, providing search, content extraction, crawling, site mapping, and research capabilities. It let… | 71 | 1409 | active |
| thinh-vu/vnstock Vnstock is an open-source Python library for extracting and analyzing Vietnam stock market data, returning data as pandas DataFrames via si… | 93 | 1408 | active |
| KoalaBear84/OpenDirectoryDownloader A cross-platform C#/.NET command-line tool that indexes open directory listings across 130+ supported formats, including FTP(S), Google Dri… | 100 | 1405 | active |
| nottelabs/reverse-api-engineer Reverse API Engineer is a Python CLI tool that captures browser network traffic (HAR) from a website and uses a configured AI model to gene… | 81 | 1404 | active |
| Maasea/sgmodule A collection of Surge modules (sgmodule files) written in JavaScript that enhance or modify mobile apps and network behavior, such as remov… | 70 | 1395 | active |
| okfn-brasil/querido-diario Querido Diário is an open-source project by Open Knowledge Brasil that scrapes and aggregates Brazilian municipal official gazettes (diário… | 75 | 1391 | active |
| mattsse/chromiumoxide chromiumoxide is a Rust library providing a high-level async API for controlling Chrome or Chromium via the Chrome DevTools Protocol. It ca… | 69 | 1389 | active |
| foxhui/WebAI2API WebAI2API is a self-hosted Node.js service that exposes web-based AI services (LMArena, Gemini, ChatGPT, DeepSeek, etc.) as OpenAI-compatib… | 54 | 1388 | active |
| Avnsx/fansly-downloader A Python-based tool for bulk downloading photos, videos, and audio from fansly.com, also shipped as a standalone Windows executable. It sup… | 10 | 1386 | active |
| lorey/mlscraper mlscraper is a Python library that automatically extracts structured data from HTML pages using machine learning. Instead of writing CSS se… | 23 | 1385 | active |
| xjbeta/iina-plus IINA+ is a small macOS application that adds danmaku (bullet comment) support and live-stream/video playback for Chinese platforms to the I… | 97 | 1384 | active |
| CIRCL/AIL-framework AIL framework is an open-source Python platform for collecting, crawling, processing, and analyzing unstructured data from the clear web, T… | 67 | 1384 | active |
| fasnow/fine Fine is a Chinese-language cyberspace asset mapping and reconnaissance tool integrating FOFA, Hunter, Quake, ZoomEye, and Shodan APIs, plus… | 77 | 1383 | active |
| jachinlin/geektime_dl A Python CLI tool that downloads Geektime (极客时间) courses and converts them into ebooks for reading on Kindle. It handles login, course quer… | 62 | 1382 | active |
| Minerchu/dongguaTV A self-hosted Node.js video aggregation platform that searches 30+ movie/TV resource site APIs, aggregates results, and scrapes metadata fr… | 35 | 1379 | active |
| fugary/calibre-douban A Calibre metadata source plugin that fetches book metadata from Douban by crawling book.douban.com web pages, since Douban no longer offer… | 56 | 1376 | active |
| iszhouhua/social-media-copilot An open-source browser extension (built with WXT and TypeScript) that scrapes data from Chinese social media platforms including Xiaohongsh… | 64 | 1373 | active |
| firecrawl/open-scouts Open Scouts is an AI-powered web monitoring platform where users create automated 'scouts' that run on a schedule to search the web and sen… | 51 | 1373 | active |
| iota9star/mikan_flutter A third-party Flutter mobile client for the Mikan Project (mikanani.me), an anime/seasonal bangumi torrent subscription site. It lets users… | 96 | 1371 | active |
| SplashtopInc/winstall winstall is a free, open-source web app for browsing and searching Microsoft's Windows Package Manager (winget) repository of 14,600+ apps.… | 77 | 1371 | active |
| dethcrypto/dethcode DethCode lets you view the verified source of deployed Ethereum smart contracts in an ephemeral VS Code instance by changing an Etherscan U… | 48 | 1368 | active |
| python273/vk_api vk_api is a Python library that wraps the VKontakte (vk.com) API, simplifying authentication and API method calls for building scripts and … | 77 | 1364 | active |
| raznem/parsera Parsera is a lightweight Python library for scraping websites using LLMs, letting users define elements to extract with natural-language de… | 49 | 1363 | active |
| xiaohucode/xiangse A curated collection of video and manga source plugins (.xbs files) for the Xiangse Guige (香色闺阁) reading/media app, imported via URL. It ag… | 10 | 1363 | active |
| xlang-ai/OpenAgents OpenAgents is an open platform for using and hosting LLM-powered language agents, featuring a Data Agent for Python/SQL analysis, a Plugins… | 29 | 4863 | maintenance |
| scrapy/parsel Parsel is a BSD-licensed Python library for extracting data from HTML, XML, and JSON documents using CSS selectors, XPath expressions, JMES… | 84 | 1358 | active |
| LeetaoGoooo/RSSAid RSSAid is a Flutter-based mobile app that complements RSSHub by helping users discover and subscribe to RSS feeds from websites, similar to… | 85 | 1353 | active |
| nmdias/FeedKit FeedKit is a Swift library for parsing and generating RSS, Atom, and JSON Feed formats. It supports common namespaces like Dublin Core, Med… | 93 | 1351 | active |
| UpgradeAll UpgradeAll is a free, open-source Android app that checks for updates to installed Android apps, Magisk modules, and other software from a … | 66 | 1350 | active |
| MiniGlome/Archive.org-Downloader A Python 3 command-line script that downloads borrowable books from archive.org and Open Library and assembles them into PDF files. It requ… | 74 | 1349 | active |
| oxylabs/paid-proxy-servers A promotional GitHub repository for Oxylabs' commercial paid proxy services, covering residential, mobile, datacenter, ISP, and SOCKS5 prox… | 56 | 1348 | active |
| philippta/flyscrape Flyscrape is a standalone command-line web scraping tool written in Go that lets users write extraction logic in JavaScript with a jQuery-l… | 39 | 1347 | active |
| nonbili/NouTube NouTube is an Android and desktop application that wraps the mobile YouTube and YouTube Music web apps in a webview, adding ad blocking, ba… | 81 | 1346 | active |
| monosans/proxy-scraper-checker A fast async Rust CLI tool that scrapes HTTP, SOCKS4, and SOCKS5 proxies from arbitrary text, HTML, or JSON sources, verifies each proxy by… | 100 | 1344 | active |
| SpiderClub/weibospider A distributed web crawler for Sina Weibo (Chinese microblogging platform) built with Python, Celery, and requests. It scrapes user profiles… | 32 | 4790 | maintenance |
| lucahammer/tweetXer A JavaScript userscript that bulk-deletes all your tweets on X (Twitter) for free, using your official Data Export file. It runs in the bro… | 52 | 1342 | active |
| mvdbos/php-spider A configurable and extensible PHP web spider library for crawling websites. It supports breadth-first and depth-first traversal, URI discov… | 72 | 1340 | active |
| mariostoev/finviz An unofficial Python library for scraping FinViz.com, providing stock data, news, insider transactions, analyst price targets, and a stock … | 54 | 1334 | active |
| jmerle/competitive-companion A browser extension that parses competitive programming problems from online judges like Codeforces, AtCoder, and Kattis. It extracts test … | 90 | 1332 | active |
| taf2/curb Curb provides Ruby-language bindings for libcurl, the fully-featured client-side URL transfer library, supporting both easy and multi modes… | 76 | 1332 | active |
| yjl9903/AnimeGarden AnimeGarden is a third-party mirror and aggregation site for 動漫花園 (dmhy) anime BT torrents, offering a web UI, open REST API, RSS feeds, an… | 85 | 1331 | active |
| sockysec/Telerecon Telerecon is a Python-based OSINT reconnaissance framework for researching and investigating Telegram. It scrapes user profiles, messages, … | 28 | 1331 | active |
| tophubs/TopList TopList (今日热榜) is a self-hosted aggregation website that collects trending headlines from popular sites like Zhihu, Hupu, and V2EX. It is w… | 32 | 4732 | maintenance |
| TheBeastLT/torrentio-scraper Torrentio is a Stremio addon ecosystem that scrapes public torrent providers and serves the results as Stremio stream results. The reposito… | 75 | 1325 | active |
| cinemagoer/cinemagoer Cinemagoer (formerly IMDbPY) is a Python package for retrieving and managing IMDb data about movies, people, characters, and companies from… | 92 | 1324 | active |
| denho/faved Faved is a free, open-source, self-hostable bookmark manager with customizable nested tags, instant search, and duplicate detection, built … | 78 | 1324 | active |
| MrTuxx/SocialPwned SocialPwned is a Python-based OSINT tool that harvests emails published on Instagram, LinkedIn, and Twitter to find credential leaks via Pw… | 10 | 1324 | active |
| sardanioss/httpcloak httpcloak is a Go HTTP client library that reproduces browser-identical TLS, HTTP/2, and HTTP/3 fingerprints (JA3/JA4, Akamai, header order… | 58 | 1319 | active |
| darwin-lau/langmanus LangManus is a community-driven Python framework for building hierarchical multi-agent AI automation systems, where a supervisor coordinate… | 22 | 1317 | active |
| Fit Launcher Fit Launcher is a desktop game launcher built with Rust, Tauri, and SolidJS for downloading and playing games from FitGirl Repacks. It uses… | 72 | 1316 | active |
| codingo/VHostScan VHostScan is a Python-based virtual host scanner that discovers hidden vhosts on a web server using wordlists, reverse lookups, and catch-a… | 36 | 1312 | active |
| Bloggify/github-calendar A JavaScript library that embeds a user's GitHub contributions calendar into any web page with a single function call. It fetches contribut… | 40 | 1311 | active |
| LainsNL/OutlookRegister A Python automation tool that mass-registers Outlook/Hotmail email accounts using browser automation (Playwright or Patchright) with simula… | 64 | 1310 | active |
| DimiMikadze/orca Orca is an AI agent application for deep LinkedIn profile analysis that scrapes posts, comments, reactions, and interaction networks, then … | 76 | 1306 | active |
| dwisiswant0/go-dork go-dork is a fast command-line dork scanner written in Go that automates Google dorking across multiple search engines. It supports Google,… | 23 | 1306 | stable |
| Steamauto/Steamauto Steamauto is a free, open-source Python application that fully automates buying, selling, and delivery of CS2/CSGO skins across Steam and C… | 92 | 1305 | active |
| bookstairs/bookhunter bookhunter is a Go command-line tool for scraping and downloading ebooks from sources like Talebook, SoBooks, Telegram channels, and China'… | 50 | 1303 | active |
| yasserg/crawler4j crawler4j is an open-source web crawler library for Java that provides a simple interface for building multi-threaded web crawlers in minut… | 23 | 4619 | maintenance |
| l429609201/misaka_danmu_server A self-hosted danmaku (bullet comment) aggregation and management server written in Python, compatible with the dandanplay API specificatio… | 79 | 1300 | active |
| cclank/news-aggregator-skill A Python-based agent skill that aggregates news from 44+ sources (tech, finance, AI, international) and generates AI-summarized daily brief… | 50 | 1297 | active |
| dvcoolarun/web2pdf A Python command-line tool that converts webpages into formatted PDFs using WeasyPrint. It supports batch conversion, recursive same-domain… | 64 | 1296 | active |
| techwithtim/Price-Tracking-Web-Scraper A full-stack price tracking application that scrapes product prices (currently Amazon.ca) using Playwright and Bright Data's Scraping Brows… | 29 | 1295 | active |
| h4r5h1t/webcopilot WebCopilot is a Bash-based automation script for bug bounty reconnaissance that enumerates subdomains using multiple tools, filters paramet… | 23 | 1294 | active |
| P1-Team/AlliN AlliN is a flexible, dependency-free Python scanner designed to assist penetration testing projects, especially lateral movement and intran… | 36 | 1287 | active |
| hadynz/obsidian-kindle-plugin An Obsidian plugin that syncs Kindle notes and highlights into your vault, either by screen-scraping Amazon's Kindle Reader cloud library o… | 69 | 1285 | active |
| umutxyp/MusicBot Beatra is an advanced Discord music bot built on discord.js v14 that streams music from YouTube, Spotify, SoundCloud, and direct links with… | 67 | 1279 | active |
| JustinBeckwith/linkinator Linkinator is a broken link checker that crawls websites, local HTML files, and markdown documentation to find dead links and invalid URLs.… | 97 | 1277 | active |
| firecrawl/fire-enrich Fire Enrich is an AI-powered data enrichment web application that transforms a list of email addresses into rich company datasets, includin… | 36 | 1277 | active |
| brunosimon/my-room-in-3d A 3D interactive recreation of the author's room built with Three.js and JavaScript, viewable in the browser. It is a creative demo/portfol… | 32 | 4511 | maintenance |
| acgotaku/YAAW-for-Chrome A Chrome extension providing a web frontend dashboard for the Aria2 download manager via its JSON-RPC interface. It can intercept browser d… | 68 | 1273 | active |
| OmniBox Spider A collection of spider (scraper) sources and interfaces for the OmniBox media application, aggregated from publicly available internet info… | 59 | 1271 | active |
| daijro/browserforge BrowserForge is a Python library that generates realistic browser headers and fingerprints, mimicking real-world browser, OS, and device di… | 67 | 1270 | active |
| minsight-ai-info/AI-Search-Hub AI Search Hub is an open-source Skill that aggregates native AI search capabilities from platforms like Gemini, Grok, Doubao, and Yuanbao i… | 47 | 1269 | active |
| 0xHJK/music-dl A Python 3 command line tool that aggregates search across multiple Chinese music sites (NetEase, QQ Music, Kugou, Baidu, Xiami, Migu) and … | 35 | 4476 | maintenance |
| mvdan/xurls A Go library and CLI tool that extracts URLs from arbitrary text using regular expressions built from TLD lists. It offers Relaxed and Stri… | 63 | 1268 | active |