function: web-scraping
1985 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| lijiejie/ds_store_exp A Python CLI exploit tool that parses exposed .DS_Store files on web servers to enumerate hidden files and directories, then recursively do… | 32 | 1736 | maintenance |
| PureDark/H-Viewer An Android feed reader and image gallery app that fetches data from websites using CSS selectors and regular expressions. Sites are added v… | 23 | 1734 | maintenance |
| EASY233/Finger Finger is a Python-based red team tool that performs liveness probing and web system fingerprint detection across large asset lists, identi… | 32 | 1724 | maintenance |
| lexborisov/myhtml MyHTML is a fast, fully HTML5-conformant HTML parser written in pure C99 with no external dependencies, using threads for asynchronous pars… | 23 | 1717 | maintenance |
| Peng-YM/QuanX A collection of JavaScript scripts and a cross-platform OpenAPI for proxy apps like Quantumult X, Loon, Surge, ShadowRocket, and JSBox. It … | 32 | 1704 | maintenance |
| henson/proxypool A Golang IP proxy pool service that scrapes free proxy sources, validates them, stores them in a database, and exposes a JSON API for crawl… | 32 | 1701 | maintenance |
| YoongiKim/AutoCrawler A Python multiprocess image web crawler that downloads images from Google and Naver image search using Selenium and ChromeDriver. It is des… | 32 | 1691 | maintenance |
| mcdallas/wallstreet Wallstreet is a Python 3 library for monitoring and analyzing real-time stock and option data, with quotes sourced from Google Finance or Y… | 32 | 1690 | maintenance |
| varietywalls/variety Variety is a wallpaper manager and downloader application for Linux desktop environments. It rotates wallpapers from local folders or onlin… | 81 | 1689 | maintenance |
| getstation/desktop-app Station is an open-source Electron-based smart browser that centralizes all your web applications (Slack, Gmail, Google Drive, and 600+ oth… | 73 | 1684 | maintenance |
| threatexpress/domainhunter Domain Hunter is a Python CLI tool that finds expired or available domains with prior benign usage history via ExpiredDomains.net, then che… | 32 | 1678 | maintenance |
| th3unkn0n/TeleGram-Scraper A Python command-line tool that scrapes Telegram groups and exports all member information to CSV using the Telegram API. It also includes … | 10 | 1672 | maintenance |
| aivarsk/scrapy-proxies A Scrapy downloader middleware that routes requests through random proxies from a configurable list to avoid IP bans. It supports multiple … | 32 | 1667 | maintenance |
| fossasia/query-server A Flask-based server that proxies keyword searches across multiple search engines (Google, Bing, DuckDuckGo, Yahoo, and others) and returns… | 10 | 1667 | maintenance |
| SwapnilSoni1999/spotify-dl A NodeJS command-line utility that downloads songs, albums, playlists, podcasts and episodes from Spotify links by sourcing the audio from … | 67 | 1666 | maintenance |
| all-in-aigc/gpts-works GPTs Works is a third-party GPTs store consisting of a website for browsing and discovering custom ChatGPT GPTs, a vector-based index syste… | 23 | 1633 | maintenance |
| oyvindeh/ucss uCSS is a Node.js tool that crawls websites or local HTML files to find unused and duplicate CSS selectors, with match counts per rule. It … | 32 | 1609 | maintenance |
| und3fined/medium-unlocker A browser extension that unlocks Medium.com member-only articles by bypassing Medium's GraphQL-based paywall. It serves as a replacement fo… | 23 | 1609 | maintenance |
| gigablast/open-source-search-engine Gigablast is a distributed open source web and enterprise search engine with a built-in spider/crawler, written in C/C++ for Linux. It powe… | 32 | 1601 | maintenance |
| propublica/upton Upton is a Ruby framework that handles the repetitive parts of writing web scrapers, letting developers focus on site-specific CSS selector… | 32 | 1597 | maintenance |
| iam-abbas/Reddit-Stock-Trends A Python application that scrapes Reddit via the PRAW API to identify trending stock tickers and analyzes their performance with yfinance. … | 32 | 1590 | maintenance |
| ericyd/gdrive-copy A Google Apps Script web app that recursively copies an entire Google Drive folder, including all sub-folders and contents, into the user's… | 10 | 1581 | maintenance |
| 0xHJK/dumpall dumpall is a Python command-line tool for exploiting information disclosure vulnerabilities on web servers. It reconstructs source code fro… | 23 | 1579 | maintenance |
| vdaubry/github-awards Git Awards (formerly GitHub Awards) is a web application that ranks GitHub users by programming language and by location (city, country, wo… | 32 | 1578 | maintenance |
| lqqyt2423/wechat_spider A Node.js WeChat crawler that uses a man-in-the-middle proxy (AnyProxy) to batch-collect WeChat official account article data, including co… | 76 | 1574 | maintenance |
| v3n0m-Scanner/V3n0M-Scanner V3n0M is an offensive security framework and vulnerability scanner written in Python 3.6+ using asyncio. It scans for SQLi, XSS, LFI/RFI vu… | 23 | 1573 | maintenance |
| ExpDev07/coronavirus-tracker-api A fast REST API built with Python and FastAPI that provides up-to-date global COVID-19 statistics including confirmed cases, deaths, and re… | 23 | 1569 | maintenance |
| kaich/Yuedu Yuedu (源阅读) is an iOS e-book reader app that lets users define custom book sources to parse and read web novels and online content. It is t… | 32 | 1564 | maintenance |
| github/lightcrawler Lightcrawler is a Node.js CLI tool that crawls a website by following links and runs each discovered page through Google Lighthouse audits.… | 10 | 1564 | maintenance |
| th3unkn0n/osi.ig A Python CLI tool that gathers OSINT information about Instagram accounts, including profile details, tags, mentions, emails, and post meta… | 32 | 1552 | maintenance |
| headzoo/surf Surf is a Go library that implements a stateful virtual web browser controlled programmatically. It supports cookies, history, bookmarks, u… | 23 | 1544 | maintenance |
| FOSSASIA-Web/api.fossasia.net FOSSASIA API is a decentralized metadata API for open source communities and hackspaces across Asia, based on the Hackerspace/Freifunk API … | 32 | 1536 | maintenance |
| tonquer/ehentai-qt A cross-platform desktop client for browsing, searching, viewing, and downloading galleries from e-hentai and exhentai, built with Python a… | 10 | 1526 | maintenance |
| CTF-MissFeng/bayonet Bayonet is a self-hosted web-based IT asset management and attack-surface platform for penetration testers, integrating subdomain enumerati… | 23 | 1517 | maintenance |
| chaitin/rad Rad (Radium) is a browser-based web crawler built for security scanning, driving a real Chrome browser to discover URLs and requests across… | 23 | 1514 | maintenance |
| mlouielu/twstock A Python library and CLI tool for fetching Taiwan stock price data from the Taiwan Stock Exchange (TWSE) and Taipei Exchange (TPEX), includ… | 88 | 1510 | maintenance |
| VideoData/DY-Data A collection of Douyin (Chinese TikTok) scraping tools and API source code covering search, user, video, live stream, comments, danmaku, an… | 32 | 1510 | maintenance |
| stephomi/sculptgl SculptGL is a browser-based 3D digital sculpting application built with WebGL, allowing users to sculpt and paint 3D meshes directly in the… | 10 | 1506 | maintenance |
| fossasia/event-collect A Python CLI tool that scrapes event website listings (e.g., EventBrite search results) and converts them into the Open Event JSON format. … | 10 | 1505 | maintenance |
| fossasia/geolocation A small FOSSASIA web application that lets users look up their geolocation coordinates via a 'Get my geolocation' button and copy them for … | 32 | 1503 | maintenance |
| duckdegen/apebot A TypeScript trading bot that detects new Binance token listings via a proxy-rotated crawler and aggressively buys the token onchain using … | 32 | 1502 | maintenance |
| fluttercommunity/flutter_webview_plugin A Flutter community plugin that lets Flutter apps communicate with a native WebView on Android and iOS, providing a WebviewScaffold widget … | 23 | 1492 | maintenance |
| eyecatchup/SEOstats SEOstats is an open-source PHP library for retrieving SEO-relevant website metrics from sources like Alexa, Google, Moz, SEMRush, Sistrix, … | 10 | 1489 | maintenance |
| FOSSASIA-Web/calendar.fossasia.org The FOSSASIA events calendar website (calendar.fossasia.org), a static site listing FOSSASIA community events, meetups, and conferences acr… | 32 | 1485 | maintenance |
| midudev/kings-league-project An educational open-source project providing a REST API and website for the Kings League Infojobs football competition, built with JavaScri… | 31 | 1482 | maintenance |
| 78/ssbc The source code of the Shousibaocai (手撕包菜) website, a self-hosted BitTorrent magnet link search engine. It combines a Node.js DHT network s… | 32 | 1476 | maintenance |
| google/flutter-desktop-embedding A collection of experimental, early-stage desktop plugins for Flutter, originally home to the Windows, macOS, and Linux Flutter embeddings … | 10 | 7046 | experimental |
| jordanpotti/AWSBucketDump AWSBucketDump is a Python CLI security tool that enumerates AWS S3 buckets using wordlists, similar to a subdomain bruteforcer but for S3. … | 32 | 1473 | maintenance |
| AlexCSDev/PatreonDownloader A command-line application for downloading content posted by creators on patreon.com, including posts, attachments, descriptions, and embed… | 69 | 1471 | maintenance |
| rotemweiss57/gpt-newspaper GPT Newspaper is a Python application built on LangGraph that runs a multi-agent pipeline to search, curate, write, critique, design, and p… | 26 | 1469 | maintenance |
| twelvesec/gasmask GasMasK is an all-in-one open source OSINT and reconnaissance tool written in Python 3. It aggregates information about a target domain fro… | 23 | 1462 | maintenance |
| udacimak/udacimak Udacimak is a command-line tool that downloads Udacity Nanodegree and course content, including videos and materials, and renders them loca… | 32 | 1457 | maintenance |
| Miserlou/SoundScrape SoundScrape is a Python command-line tool that downloads tracks, sets, groups, and likes from SoundCloud, as well as albums from Bandcamp a… | 32 | 1446 | maintenance |
| liguobao/HouseSearch A map-driven rental housing aggregation platform that continuously crawls public rental listings from sources like Douban, Beike, and Xiaoh… | 76 | 1439 | maintenance |
| xisuo67/XHS-Spider XHS-Spider is a polished Windows desktop (WPF, .NET 6) tool for collecting Xiaohongshu (Little Red Book) data, including keyword and user s… | 64 | 1435 | maintenance |
| wummel/linkchecker LinkChecker is a GPL-licensed Python tool that checks links in web documents or crawls entire websites to find broken links. It supports re… | 32 | 1435 | maintenance |
| pushshift/api Pushshift is a RESTful API service for searching Reddit's historical comments and submissions, backed by Elasticsearch and full data archiv… | 32 | 1433 | maintenance |
| damklis/DataEngineeringProject An end-to-end data engineering project that scrapes news from RSS feeds via Airflow-scheduled Python scrapers and streams them through Kafk… | 32 | 1429 | maintenance |
| jonnnnyw/php-phantomjs A PHP library that wraps the PhantomJS headless browser, letting PHP applications load web pages with full JavaScript support and inspect t… | 23 | 1426 | maintenance |
| mckaywrigley/clarity-ai Clarity AI is an open-source clone of Perplexity.ai that answers user queries by scraping Google search results, parsing the webpages, and … | 31 | 1418 | maintenance |
| chiphuyen/sotawhat A Python CLI script that crawls arXiv for papers matching a keyword and returns summarized abstracts with title, author, date, and link. It… | 23 | 1418 | maintenance |
| mps-youtube/pafy Pafy is a Python library for retrieving YouTube video metadata and downloading video or audio streams at requested resolutions and formats.… | 23 | 1416 | maintenance |
| OreosLab/checkinpanel A collection of automated check-in (sign-in) scripts for various websites, designed to run on scheduled task panels like elecV2P and Qinglo… | 32 | 1402 | maintenance |
| zhuweiyou/weixin-game-helper A collection of helper bots and automation tools for WeChat mini games such as Jump Jump, Head Master, and drawing-guessing games. It captu… | 10 | 1400 | maintenance |
| idevelop/ascii-camera A browser-based web app that converts your webcam video stream into a real-time ASCII art representation using the HTML5 getUserMedia API. … | 32 | 1399 | maintenance |
| maK-/parameth parameth is a Python command-line tool that brute-forces GET and POST parameter names on web endpoints by comparing response differences. I… | 10 | 1396 | maintenance |
| sham00n/buster Buster is a Python command-line OSINT tool for email reconnaissance. It finds social accounts linked to an email, data breaches, pastes, re… | 32 | 1395 | maintenance |
| kotartemiy/pygooglenews A Python wrapper around the Google News RSS feed providing top stories, topic and geolocation feeds, and full-text search with date-range a… | 32 | 1391 | maintenance |
| loklak Loklak is a distributed Java server application that anonymously collects, indexes, and shares social media messages (especially tweets) us… | 23 | 1391 | maintenance |
| jaysonlong/webvideo-downloader A two-part web video downloader: a browser userscript (Violentmonkey/Tampermonkey) extracts video stream URLs from supported sites, and a P… | 32 | 1384 | maintenance |
| xalanq/cf-tool A small, cross-platform command-line interface for Codeforces written in Go. It lets competitive programmers race contests, fetch samples, … | 10 | 1384 | maintenance |
| amirgamil/apollo Apollo is a Unix-style personal search engine and web crawler for indexing your own digital footprint. You manually curate what goes into i… | 32 | 1380 | maintenance |
| LaurenceRawlings/savify Savify is a Python library and CLI tool that downloads songs from providers like YouTube and embeds metadata scraped from Spotify, producin… | 23 | 1377 | maintenance |
| BobbyESP/Spowlo Spowlo is an Android app for downloading Spotify songs and playlists, powered by the spotDL Python library and built with Jetpack Compose a… | 10 | 1371 | maintenance |
| sourcegraph/webloop WebLoop is a scriptable, headless WebKit browser with a Go API, comparable to PhantomJS but for Go. It can render static HTML from dynamic … | 32 | 1369 | maintenance |
| PKUJohnson/OpenData OpenDataTools is a Python library that scrapes financial and investment data from various websites and exposes it through simple, easy-to-u… | 32 | 1367 | maintenance |
| martinsbalodis/web-scraper-chrome-extension Web Scraper is a Chrome browser extension for extracting data from web pages without writing code. Users define sitemaps describing how to … | 32 | 1363 | maintenance |
| felipecsl/wombat Wombat is a lightweight Ruby web crawler and scraper library with an elegant DSL for extracting structured data from web pages. It lets dev… | 73 | 1360 | maintenance |
| ai-to-ai/Auto-Gmail-Creator A Python Selenium-based bot that bulk-creates Gmail accounts automatically, using sms-activate.org for phone verification and webdriver-man… | 31 | 1360 | maintenance |
| GoogleChromeLabs/proxx PROXX is an open-source browser-based Minesweeper-style proximity game built with Preact, WebGL/Canvas rendering, and web workers. It is de… | 62 | 1359 | maintenance |
| huaying/instagram-crawler A Python command-line crawler that scrapes Instagram posts, profiles, and hashtag data using Selenium and ChromeDriver, without the officia… | 32 | 1350 | maintenance |
| adriancooney/puppeteer-heap-snapshot A TypeScript library and CLI for capturing Chrome DevTools heap snapshots from Puppeteer-controlled pages and querying them for objects wit… | 23 | 1350 | maintenance |
| square/square.github.io A static Jekyll-based portal website that showcases Square's open source projects, styled like a Square merchant directory page. Repository… | 32 | 1341 | maintenance |
| scrapinghub/frontera Frontera is a Python web crawling framework that implements a scalable crawl frontier, storing and prioritizing links extracted by crawlers… | 34 | 1332 | maintenance |
| w-digital-scanner/w12scan w12scan is a self-hosted network asset discovery engine that aggregates scan results into a searchable web interface backed by Elasticsearc… | 23 | 1331 | maintenance |
| bugswriter/tuxi Tuxi is a shell-based CLI assistant that scrapes Google search results and SERPs to return instant, concise answers to natural language que… | 10 | 1331 | maintenance |
| laramies/metagoofil Metagoofil is a Python command-line OSINT tool that searches Google for public documents (pdf, doc, xls, ppt) on target websites, downloads… | 32 | 1313 | maintenance |
| heremaps/harp.gl harp.gl is an open-source 3D map rendering engine written in TypeScript and built on three.js/WebGL. It enables themeable, high-performance… | 10 | 1313 | maintenance |
| danielcardeenas/sulla Sulla is a JavaScript/TypeScript library providing a high-level API to control WhatsApp via Puppeteer, enabling automated responses and mes… | 23 | 1308 | maintenance |
| c4tcom/Katana Katana-ds is a Python CLI tool that automates advanced Google queries known as Google Dorks (Google Dorking), with optional Tor support for… | 10 | 1304 | maintenance |
| InstaPy/instagram-profilecrawl A Python script from the InstaPy ecosystem that crawls public Instagram profile information such as post counts, follower counts, and post … | 32 | 1301 | maintenance |
| devanshbatham/FavFreak A Python CLI tool that fetches favicon.ico files from lists of URLs, computes their mmh3 hashes, and groups domains/subdomains/IPs by match… | 23 | 1298 | maintenance |
| LangziFun/LangSrcCurise LangSrcCurise is a Django-based automated subdomain asset monitoring system for security researchers tracking SRC (Security Response Center… | 32 | 1294 | maintenance |
| honzaap/GithubCity A web application that generates a 3D city from your GitHub contribution graph using Three.js. Buildings represent contributions, letting u… | 32 | 1293 | maintenance |
| manu354/cryptocurrency-arbitrage A Node.js cryptocurrency arbitrage opportunity calculator that scans over 800 currencies across 50 markets, comparing prices to find the hi… | 23 | 1282 | maintenance |
| sunra/php-simple-html-dom-parser A PHP library adapting Simple HTML DOM Parser for Composer and PSR-0, allowing easy HTML parsing and manipulation. It supports invalid HTML… | 23 | 1281 | maintenance |
| marcelja/facebook-delete A Go CLI tool that bulk-deletes Facebook activity (posts, comments, likes, etc.) without deleting the whole account, using rate-limited req… | 10 | 1281 | maintenance |
| Sniper970119/dianping_spider A Python web scraper for Dianping (大众点评) that extracts search results, shop details, and reviews, handling dynamic font encryption without … | 72 | 1278 | maintenance |
| Song-Li/cross_browser A research project from Johns Hopkins implementing cross-browser fingerprinting that identifies users across different browsers on the same… | 32 | 1270 | maintenance |
| goldfishh/chatgpt-tool-hub An open-source Python tool engine that lets ChatGPT (and other LLMs) invoke tools like web search, math computation, code execution, and co… | 21 | 1267 | maintenance |