Open-source web-scraping software
2000 open-source libraries, tools and applications tagged web-scraping, ranked by how closely web-scraping is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
2000 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| megadose/OnionSearch OnionSearch is a Python 3 CLI script that scrapes URLs from multiple .onion search engines such as Ahmia, Phobos, and Deeplink. It supports… | 32 | 1825 | maintenance |
| Yvesssn/DetectDee DetectDee is a Go CLI tool for OSINT that hunts down social media accounts by username, email, or phone number across many social networks.… | 21 | 1824 | maintenance |
| kickscondor/fraidycat Fraidycat is an app (and browser extension) for following people across many platforms - blogs, wikis, YouTube, Twitter, Reddit, Instagram … | 23 | 1823 | maintenance |
| chaychan/TouTiao An Android app that closely replicates the Toutiao (Today's Headlines) news client, built with RxJava, Retrofit, and MVP architecture. It s… | 32 | 1816 | maintenance |
| vertex-app/vertex Vertex is a self-hosted management tool for private tracker (PT) users that combines TV-show tracking with automated torrent snatching ('sh… | 76 | 1802 | maintenance |
| anvaka/pm Software Galaxies is an interactive web visualization that renders major software package manager ecosystems (npm, Bower, Composer, RubyGem… | 69 | 1789 | maintenance |
| 1tayH/noisy A Python script that generates random HTTP and DNS traffic noise in the background to obscure your real browsing patterns. It crawls config… | 32 | 1780 | maintenance |
| htmlpreview/htmlpreview.github.com A client-side web tool that renders HTML files hosted on GitHub or BitBucket repositories without cloning or downloading them. It prepends … | 32 | 1771 | maintenance |
| reworkd/tarsier Tarsier is a Python library providing vision utilities for LLM-driven web interaction agents. It visually tags interactable page elements w… | 18 | 1767 | maintenance |
| intoli/remote-browser Remote Browser is a JavaScript library for programmatically controlling browsers like Chrome and Firefox, built on the standard Web Extensi… | 32 | 1751 | maintenance |
| BKcore/HexGL HexGL is a futuristic, fast-paced HTML5 racing game built with JavaScript and WebGL using three.js, inspired by Wipeout and F-Zero. This re… | 32 | 1746 | maintenance |
| DanMcInerney/xsscrapy A Python-based spider built on Scrapy that crawls a website and tests every link it finds for cross-site scripting (XSS) and basic SQL inje… | 32 | 1741 | maintenance |
| iqiqiya/iqiqiya-API A PHP-based collection of free web API endpoints for parsing media from Chinese platforms (Douyin, Kuaishou, Bilibili, NetEase Music, Ximal… | 10 | 1741 | maintenance |
| howie6879/ruia Ruia is an async web scraping micro-framework for Python 3.6+ built on asyncio and aiohttp. It offers declarative Item/Field extraction (XP… | 67 | 1739 | maintenance |
| lijiejie/ds_store_exp A Python CLI exploit tool that parses exposed .DS_Store files on web servers to enumerate hidden files and directories, then recursively do… | 32 | 1739 | maintenance |
| HN Reader (SwiftHN) SwiftHN is an open-source Hacker News reader iOS app written in Swift, published on the App Store as HN Reader. It uses its own scraping li… | 32 | 1737 | maintenance |
| PureDark/H-Viewer An Android feed reader and image gallery app that fetches data from websites using CSS selectors and regular expressions. Sites are added v… | 23 | 1732 | maintenance |
| EASY233/Finger Finger is a Python-based red team tool that performs liveness probing and web system fingerprint detection across large asset lists, identi… | 32 | 1725 | maintenance |
| Modest engine MyHTML is a fast, fully HTML5-conformant HTML parser written in pure C99 with no external dependencies, using threads for asynchronous pars… | 23 | 1716 | maintenance |
| mcdallas/wallstreet Wallstreet is a Python 3 library for monitoring and analyzing real-time stock and option data, with quotes sourced from Google Finance or Y… | 32 | 1713 | maintenance |
| varietywalls/variety Variety is a wallpaper manager and downloader application for Linux desktop environments. It rotates wallpapers from local folders or onlin… | 87 | 1711 | maintenance |
| Peng-YM/QuanX A collection of JavaScript scripts and a cross-platform OpenAPI for proxy apps like Quantumult X, Loon, Surge, ShadowRocket, and JSBox. It … | 32 | 1705 | maintenance |
| henson/proxypool A Golang IP proxy pool service that scrapes free proxy sources, validates them, stores them in a database, and exposes a JSON API for crawl… | 32 | 1701 | maintenance |
| YoongiKim/AutoCrawler A Python multiprocess image web crawler that downloads images from Google and Naver image search using Selenium and ChromeDriver. It is des… | 74 | 1694 | maintenance |
| getstation/desktop-app Station is an open-source Electron-based smart browser that centralizes all your web applications (Slack, Gmail, Google Drive, and 600+ oth… | 67 | 1683 | maintenance |
| threatexpress/domainhunter Domain Hunter is a Python CLI tool that finds expired or available domains with prior benign usage history via ExpiredDomains.net, then che… | 32 | 1679 | maintenance |
| SwapnilSoni1999/spotify-dl A NodeJS command-line utility that downloads songs, albums, playlists, podcasts and episodes from Spotify links by sourcing the audio from … | 66 | 1674 | maintenance |
| th3unkn0n/TeleGram-Scraper A Python command-line tool that scrapes Telegram groups and exports all member information to CSV using the Telegram API. It also includes … | 10 | 1672 | maintenance |
| aivarsk/scrapy-proxies A Scrapy downloader middleware that routes requests through random proxies from a configurable list to avoid IP bans. It supports multiple … | 32 | 1666 | maintenance |
| fossasia/query-server A Flask-based server that proxies keyword searches across multiple search engines (Google, Bing, DuckDuckGo, Yahoo, and others) and returns… | 10 | 1666 | maintenance |
| all-in-aigc/gpts-works GPTs Works is a third-party GPTs store consisting of a website for browsing and discovering custom ChatGPT GPTs, a vector-based index syste… | 20 | 1629 | maintenance |
| und3fined/medium-unlocker A browser extension that unlocks Medium.com member-only articles by bypassing Medium's GraphQL-based paywall. It serves as a replacement fo… | 23 | 1614 | maintenance |
| oyvindeh/ucss uCSS is a Node.js tool that crawls websites or local HTML files to find unused and duplicate CSS selectors, with match counts per rule. It … | 32 | 1606 | maintenance |
| gigablast/open-source-search-engine Gigablast is a distributed open source web and enterprise search engine with a built-in spider/crawler, written in C/C++ for Linux. It powe… | 32 | 1603 | maintenance |
| propublica/upton Upton is a Ruby framework that handles the repetitive parts of writing web scrapers, letting developers focus on site-specific CSS selector… | 32 | 1594 | maintenance |
| iam-abbas/Reddit-Stock-Trends A Python application that scrapes Reddit via the PRAW API to identify trending stock tickers and analyzes their performance with yfinance. … | 32 | 1592 | maintenance |
| lqqyt2423/wechat_spider A Node.js WeChat crawler that uses a man-in-the-middle proxy (AnyProxy) to batch-collect WeChat official account article data, including co… | 72 | 1591 | maintenance |
| ericyd/gdrive-copy A Google Apps Script web app that recursively copies an entire Google Drive folder, including all sub-folders and contents, into the user's… | 10 | 1579 | maintenance |
| 0xHJK/dumpall dumpall is a Python command-line tool for exploiting information disclosure vulnerabilities on web servers. It reconstructs source code fro… | 23 | 1578 | maintenance |
| v3n0m-Scanner/V3n0M-Scanner V3n0M is an offensive security framework and vulnerability scanner written in Python 3.6+ using asyncio. It scans for SQLi, XSS, LFI/RFI vu… | 23 | 1578 | maintenance |
| vdaubry/github-awards Git Awards (formerly GitHub Awards) is a web application that ranks GitHub users by programming language and by location (city, country, wo… | 32 | 1576 | maintenance |
| th3unkn0n/osi.ig A Python CLI tool that gathers OSINT information about Instagram accounts, including profile details, tags, mentions, emails, and post meta… | 32 | 1572 | maintenance |
| kaich/Yuedu Yuedu (源阅读) is an iOS e-book reader app that lets users define custom book sources to parse and read web novels and online content. It is t… | 32 | 1570 | maintenance |
| github/lightcrawler Lightcrawler is a Node.js CLI tool that crawls a website by following links and runs each discovered page through Google Lighthouse audits.… | 10 | 1569 | maintenance |
| ExpDev07/coronavirus-tracker-api A fast REST API built with Python and FastAPI that provides up-to-date global COVID-19 statistics including confirmed cases, deaths, and re… | 23 | 1568 | maintenance |
| VideoData/DY-Data A collection of Douyin (Chinese TikTok) scraping tools and API source code covering search, user, video, live stream, comments, danmaku, an… | 32 | 1548 | maintenance |
| headzoo/surf Surf is a Go library that implements a stateful virtual web browser controlled programmatically. It supports cookies, history, bookmarks, u… | 23 | 1544 | maintenance |
| FOSSASIA-Web/api.fossasia.net FOSSASIA API is a decentralized metadata API for open source communities and hackspaces across Asia, based on the Hackerspace/Freifunk API … | 32 | 1534 | maintenance |
| tonquer/ehentai-qt A cross-platform desktop client for browsing, searching, viewing, and downloading galleries from e-hentai and exhentai, built with Python a… | 10 | 1532 | maintenance |
| bayonet Bayonet is a self-hosted web-based IT asset management and attack-surface platform for penetration testers, integrating subdomain enumerati… | 23 | 1524 | maintenance |
| mlouielu/twstock A Python library and CLI tool for fetching Taiwan stock price data from the Taiwan Stock Exchange (TWSE) and Taipei Exchange (TPEX), includ… | 82 | 1520 | maintenance |
| chaitin/rad Rad (Radium) is a browser-based web crawler built for security scanning, driving a real Chrome browser to discover URLs and requests across… | 23 | 1519 | maintenance |
| duckdegen/apebot A TypeScript trading bot that detects new Binance token listings via a proxy-rotated crawler and aggressively buys the token onchain using … | 32 | 1506 | maintenance |
| stephomi/sculptgl SculptGL is a browser-based 3D digital sculpting application built with WebGL, allowing users to sculpt and paint 3D meshes directly in the… | 10 | 1505 | maintenance |
| fossasia/event-collect A Python CLI tool that scrapes event website listings (e.g., EventBrite search results) and converts them into the Open Event JSON format. … | 10 | 1503 | maintenance |
| fossasia/geolocation A small FOSSASIA web application that lets users look up their geolocation coordinates via a 'Get my geolocation' button and copy them for … | 32 | 1501 | maintenance |
| fluttercommunity/flutter_webview_plugin A Flutter community plugin that lets Flutter apps communicate with a native WebView on Android and iOS, providing a WebviewScaffold widget … | 23 | 1495 | maintenance |
| AlexCSDev/PatreonDownloader A command-line application for downloading content posted by creators on patreon.com, including posts, attachments, descriptions, and embed… | 63 | 1494 | maintenance |
| eyecatchup/SEOstats SEOstats is an open-source PHP library for retrieving SEO-relevant website metrics from sources like Alexa, Google, Moz, SEMRush, Sistrix, … | 10 | 1491 | maintenance |
| FOSSASIA-Web/calendar.fossasia.org The FOSSASIA events calendar website (calendar.fossasia.org), a static site listing FOSSASIA community events, meetups, and conferences acr… | 32 | 1482 | maintenance |
| twelvesec/gasmask GasMasK is an all-in-one open source OSINT and reconnaissance tool written in Python 3. It aggregates information about a target domain fro… | 23 | 1480 | maintenance |
| midudev/kings-league-project An educational open-source project providing a REST API and website for the Kings League Infojobs football competition, built with JavaScri… | 10 | 1479 | maintenance |
| 78/ssbc The source code of the Shousibaocai (手撕包菜) website, a self-hosted BitTorrent magnet link search engine. It combines a Node.js DHT network s… | 32 | 1476 | maintenance |
| liguobao/HouseSearch A map-driven rental housing aggregation platform that continuously crawls public rental listings from sources like Douban, Beike, and Xiaoh… | 77 | 1475 | maintenance |
| google/flutter-desktop-embedding A collection of experimental, early-stage desktop plugins for Flutter, originally home to the Windows, macOS, and Linux Flutter embeddings … | 10 | 7036 | experimental |
| rotemweiss57/gpt-newspaper GPT Newspaper is a Python application built on LangGraph that runs a multi-agent pipeline to search, curate, write, critique, design, and p… | 26 | 1472 | maintenance |
| jordanpotti/AWSBucketDump AWSBucketDump is a Python CLI security tool that enumerates AWS S3 buckets using wordlists, similar to a subdomain bruteforcer but for S3. … | 32 | 1471 | maintenance |
| udacimak/udacimak Udacimak is a command-line tool that downloads Udacity Nanodegree and course content, including videos and materials, and renders them loca… | 32 | 1460 | maintenance |
| Miserlou/SoundScrape SoundScrape is a Python command-line tool that downloads tracks, sets, groups, and likes from SoundCloud, as well as albums from Bandcamp a… | 32 | 1447 | maintenance |
| xisuo67/XHS-Spider XHS-Spider is a polished Windows desktop (WPF, .NET 6) tool for collecting Xiaohongshu (Little Red Book) data, including keyword and user s… | 59 | 1443 | maintenance |
| damklis/DataEngineeringProject An end-to-end data engineering project that scrapes news from RSS feeds via Airflow-scheduled Python scrapers and streams them through Kafk… | 32 | 1440 | maintenance |
| wummel/linkchecker LinkChecker is a GPL-licensed Python tool that checks links in web documents or crawls entire websites to find broken links. It supports re… | 32 | 1434 | maintenance |
| pushshift/api Pushshift is a RESTful API service for searching Reddit's historical comments and submissions, backed by Elasticsearch and full data archiv… | 32 | 1434 | maintenance |
| jonnnnyw/php-phantomjs A PHP library that wraps the PhantomJS headless browser, letting PHP applications load web pages with full JavaScript support and inspect t… | 10 | 1425 | maintenance |
| mckaywrigley/clarity-ai Clarity AI is an open-source clone of Perplexity.ai that answers user queries by scraping Google search results, parsing the webpages, and … | 31 | 1423 | maintenance |
| chiphuyen/sotawhat A Python CLI script that crawls arXiv for papers matching a keyword and returns summarized abstracts with title, author, date, and link. It… | 23 | 1421 | maintenance |
| sham00n/buster Buster is a Python command-line OSINT tool for email reconnaissance. It finds social accounts linked to an email, data breaches, pastes, re… | 32 | 1420 | maintenance |
| mps-youtube/pafy Pafy is a Python library for retrieving YouTube video metadata and downloading video or audio streams at requested resolutions and formats.… | 23 | 1416 | maintenance |
| idevelop/ascii-camera A browser-based web app that converts your webcam video stream into a real-time ASCII art representation using the HTML5 getUserMedia API. … | 32 | 1403 | maintenance |
| ai-to-ai/Auto-Gmail-Creator A Python Selenium-based bot that bulk-creates Gmail accounts automatically, using sms-activate.org for phone verification and webdriver-man… | 31 | 1402 | maintenance |
| zhuweiyou/weixin-game-helper A collection of helper bots and automation tools for WeChat mini games such as Jump Jump, Head Master, and drawing-guessing games. It captu… | 10 | 1401 | maintenance |
| OreosLab/checkinpanel A collection of automated check-in (sign-in) scripts for various websites, designed to run on scheduled task panels like elecV2P and Qinglo… | 32 | 1399 | maintenance |
| LaurenceRawlings/savify Savify is a Python library and CLI tool that downloads songs from providers like YouTube and embeds metadata scraped from Spotify, producin… | 23 | 1399 | maintenance |
| kotartemiy/pygooglenews A Python wrapper around the Google News RSS feed providing top stories, topic and geolocation feeds, and full-text search with date-range a… | 32 | 1396 | maintenance |
| maK-/parameth parameth is a Python command-line tool that brute-forces GET and POST parameter names on web endpoints by comparing response differences. I… | 10 | 1396 | maintenance |
| jaysonlong/webvideo-downloader A two-part web video downloader: a browser userscript (Violentmonkey/Tampermonkey) extracts video stream URLs from supported sites, and a P… | 32 | 1393 | maintenance |
| loklak Loklak is a distributed Java server application that anonymously collects, indexes, and shares social media messages (especially tweets) us… | 23 | 1393 | maintenance |
| amirgamil/apollo Apollo is a Unix-style personal search engine and web crawler for indexing your own digital footprint. You manually curate what goes into i… | 32 | 1380 | maintenance |
| xalanq/cf-tool A small, cross-platform command-line interface for Codeforces written in Go. It lets competitive programmers race contests, fetch samples, … | 10 | 1377 | maintenance |
| BobbyESP/Spowlo Spowlo is an Android app for downloading Spotify songs and playlists, powered by the spotDL Python library and built with Jetpack Compose a… | 10 | 1370 | maintenance |
| PKUJohnson/OpenData OpenDataTools is a Python library that scrapes financial and investment data from various websites and exposes it through simple, easy-to-u… | 32 | 1369 | maintenance |
| WebLoop WebLoop is a scriptable, headless WebKit browser with a Go API, comparable to PhantomJS but for Go. It can render static HTML from dynamic … | 32 | 1368 | maintenance |
| martinsbalodis/web-scraper-chrome-extension Web Scraper is a Chrome browser extension for extracting data from web pages without writing code. Users define sitemaps describing how to … | 32 | 1364 | maintenance |
| huaying/instagram-crawler A Python command-line crawler that scrapes Instagram posts, profiles, and hashtag data using Selenium and ChromeDriver, without the officia… | 32 | 1362 | maintenance |
| felipecsl/wombat Wombat is a lightweight Ruby web crawler and scraper library with an elegant DSL for extracting structured data from web pages. It lets dev… | 67 | 1360 | maintenance |
| GoogleChromeLabs/proxx PROXX is an open-source browser-based Minesweeper-style proximity game built with Preact, WebGL/Canvas rendering, and web workers. It is de… | 10 | 1358 | maintenance |
| square/square.github.io A static Jekyll-based portal website that showcases Square's open source projects, styled like a Square merchant directory page. Repository… | 32 | 1356 | maintenance |
| adriancooney/puppeteer-heap-snapshot A TypeScript library and CLI for capturing Chrome DevTools heap snapshots from Puppeteer-controlled pages and querying them for objects wit… | 23 | 1347 | maintenance |
| scrapinghub/frontera Frontera is a Python web crawling framework that implements a scalable crawl frontier, storing and prioritizing links extracted by crawlers… | 30 | 1334 | maintenance |
| laramies/metagoofil Metagoofil is a Python command-line OSINT tool that searches Google for public documents (pdf, doc, xls, ppt) on target websites, downloads… | 32 | 1333 | maintenance |