Open-source web-scraping software
2000 open-source libraries, tools and applications tagged web-scraping, ranked by how closely web-scraping is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
2000 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| fossasia/loklak-webtweets A small client-side web app that fetches and displays tweets via the loklak API, embeddable on any website with configurable query, count, … | 10 | 1525 | abandoned |
| GravityLabs/goose Goose is a Scala library (originally Java) that extracts the main body text, metadata, publish date, embedded videos, and top image from ne… | 10 | 1524 | abandoned |
| fossasia/loklak_tweetheatmap A web application that displays a geographic heat map of tweets matching a search query, built with Angular.js and OpenLayers 3 using the L… | 32 | 1517 | abandoned |
| githubwing/GankClient-Kotlin An Android client app for gank.io (a Chinese tech content aggregator) written in Kotlin with Material Design. It serves as a reference impl… | 32 | 1515 | abandoned |
| yhat/scrape A Go library providing a higher-level interface over golang.org/x/net/html for web scraping. It offers generic tree traversal helpers like … | 10 | 1513 | abandoned |
| FOSSASIA-Web/timeline.api.fossasia.net A jQuery plugin that embeds a FOSSASIA community event timeline into any website, fetching events from the FOSSASIA/Freifunk Calendar API o… | 32 | 1512 | abandoned |
| isanchop/stuhack A Chrome extension that attempted to unlock Studocu premium features such as removing premium banners, bypassing document blur, and enablin… | 10 | 1506 | abandoned |
| fossasia/wp-tweet-legacy A WordPress widget plugin that displays Twitter feeds in widget-ready areas, parsing @usernames, #hashtags, and URLs into links. It support… | 32 | 1503 | abandoned |
| qinxuye/cola Cola is a high-level distributed crawling framework in Python for scraping pages and extracting structured data from websites. The same cra… | 10 | 1499 | abandoned |
| artyshko/smd A Python application that downloads high-quality music from Spotify playlists, albums, and songs by fetching audio from sources like YouTub… | 23 | 1496 | abandoned |
| fossasia/loklak_heatmapper A plain HTML/JavaScript web application that visualizes recent tweet activity as a heatmap on a world map. It fetches tweets via the loklak… | 10 | 1495 | abandoned |
| johntitus/node-horseman A Node.js library providing a chainable, Promise-based API for controlling the PhantomJS headless browser, supporting page navigation, form… | 32 | 1480 | abandoned |
| 锤子商城 A full-stack e-commerce demo mall (clone of the Smartisan/锤子 online store) built with Vue 2 + Vuex on the frontend and Node.js + MongoDB fo… | 32 | 1480 | abandoned |
| ring04h/wydomain wydomain is a Python command-line tool for discovering subdomains of a target domain. It combines dictionary-based DNS bruteforcing with qu… | 32 | 1480 | abandoned |
| fossasia/loklak_scraper_js A collection of JavaScript scrapers for the loklak project that extract data from websites like Twitter and Quora and output JSON resemblin… | 10 | 1471 | abandoned |
| fossasia/loklak-timeline-plugin A lightweight JavaScript plugin for embedding loklak search timelines into web pages via a simple HTML tag with data attributes. It renders… | 10 | 1455 | abandoned |
| keenwon/antcolony AntColony is a Node.js-based BitTorrent DHT network crawler that collects active infohashes, downloads and parses torrent files, and stores… | 32 | 1454 | abandoned |
| EvilCult/Video-Downloader A Python-based desktop application that resolves the real video file URLs from Chinese video sites (Youku, Tudou, LeTV, Sohu, Bilibili, AcF… | 23 | 1450 | abandoned |
| Entromorgan/Autoticket A Python/Selenium automation tool that automatically snipes and purchases tickets on Damai.cn (China's major ticketing site) by driving Chr… | 10 | 1448 | abandoned |
| yask123/Instant-Music-Downloader A Python CLI tool that downloads songs from the web given a song name, lyrics snippet, or artist, and embeds ID3 metadata (title, artist, a… | 23 | 1446 | abandoned |
| jamesturk/scrapeghost scrapeghost is an experimental Python library that uses OpenAI's GPT models to scrape structured data from websites without writing page-sp… | 46 | 1439 | abandoned |
| superSearchPlus superSearchPlus is a Chrome browser extension that aggregates information gathering for white-hat hackers, integrating common asset mapping… | 32 | 1434 | abandoned |
| mafintosh/torrent-mount A Node.js CLI tool that mounts torrents or magnet links as a filesystem in real time using torrent-stream and FUSE. Files appear in a mount… | 32 | 1414 | abandoned |
| OpnTec/parliament-scraper A collection of scrapers (in Python, Ruby, and Scala) that download public parliamentary data such as written questions from the EU Parliam… | 32 | 1404 | abandoned |
| DocNow/twarc twarc is a command line tool and Python library for collecting and archiving Twitter JSON data via the Twitter v1.1 and v2 APIs. It was dev… | 44 | 1395 | abandoned |
| regenrek/deepwiki-mcp An unofficial MCP server that fetches deepwiki.com documentation, crawls its pages, converts them to Markdown, and serves them to MCP-compa… | 49 | 1386 | abandoned |
| leonardocardoso/SwiftLinkPreview A Swift library that generates link previews from URLs by extracting titles, relevant text, and images. It supports iOS, macOS, watchOS, an… | 10 | 1385 | abandoned |
| SamyPesse/tv.js A Node.js smart TV application that searches for movies via iTunes' API and isoHunt, then streams them over BitTorrent directly to a TV-con… | 32 | 1363 | abandoned |
| dinubs/jam-api Jam API is a hosted service (self-hostable Node.js app) that turns any website into a JSON API by extracting data using CSS query selectors… | 32 | 1359 | abandoned |
| Vespa314/bilibili-api A collection of Bilibili (B站) API documentation and Python tooling for scraping video, user, comment, danmaku, and bangumi data. It include… | 32 | 1357 | abandoned |
| aki66938/xhs-toolkit A Python MCP toolkit for Xiaohongshu (RED) creators that lets AI clients like Claude Desktop publish image/video notes, manage login cookie… | 57 | 1349 | abandoned |
| romannurik/LayerVisualizer A simple web-based 3D layer visualizer for visualizing material UIs and other designs involving depth and shadows. It also includes a layou… | 32 | 1348 | abandoned |
| filamentgroup/SocialCount SocialCount is a small jQuery plugin providing progressively enhanced, lazy-loaded social sharing widgets for Facebook, Twitter, and Google… | 10 | 1348 | abandoned |
| Jefferson-Henrique/GetOldTweets-python A Python library that retrieves old tweets by mimicking the JSON calls Twitter Search makes in the browser, bypassing the official API's ti… | 32 | 1339 | abandoned |
| rhiever/TwitterFollowBot A Python library that automates Twitter actions such as following users and favoriting tweets via the Twitter API. It is explicitly no long… | 23 | 1334 | abandoned |
| rialto-php/puphpeteer PuPHPeteer is a PHP library that bridges to Puppeteer via Node, exposing the full Puppeteer API for controlling headless Chrome from PHP co… | 10 | 1329 | abandoned |
| acheong08/ChatGPTProxy A Go-based proxy server that bypasses Cloudflare protection on ChatGPT using TLS spoofing, exposing ChatGPT through an API-like interface. … | 10 | 1329 | abandoned |
| wintercn/dog-fucked-zhihu A browser plugin (userscript) that helps users quit Zhihu by backing up their answers, removing all upvotes, and batch-replacing or deletin… | 32 | 1315 | abandoned |
| JD_AutoBuy A Python 2.7 command-line scraper that logs into JD.com (via QR code or credentials), monitors product stock and price, and automatically p… | 32 | 1309 | abandoned |
| Skyexu/TopSup A Python CLI assistant for live trivia quiz apps like 冲顶大会 (Top Hero) that captures Android phone screenshots via ADB, OCRs the question an… | 32 | 1302 | abandoned |
| mov-cli mov-cli is a Python CLI tool that lets users search for and stream movies and TV shows directly from the terminal, using a plugin system to… | 51 | 1299 | abandoned |
| defaultnamehere/zzzzz A Python application that continuously collects Facebook friends' activity data and generates graphs visualizing when they are online or ac… | 32 | 1298 | abandoned |
| szdc/tiktok-api An unofficial, reverse-engineered TypeScript API wrapper for TikTok (formerly Musical.ly), published as the npm package tiktok-api. It lets… | 10 | 1292 | abandoned |
| zhaoxianjin/91porn An unofficial Android client for the adult video site 91Porn, built with RxJava, Retrofit, RxCache, MVP, and Jsoup. It bypasses the guest l… | 32 | 1290 | abandoned |
| LiuRoy/zhihu_spider A Scrapy-based web crawler that scrapes Zhihu user profiles and their follower/following relationship graphs, storing data in MongoDB. It s… | 32 | 1289 | abandoned |
| hilongjw/vue-zhihu-daily A single-page web application that presents Zhihu Daily content, built with Vue.js on the frontend and Node.js/Express on the backend. It o… | 32 | 1269 | abandoned |
| MiningCattiva/x-spider X-Spider is a desktop media downloader for X (Twitter) that fetches images and videos from accounts. It supports media filters, configurabl… | 10 | 1256 | abandoned |
| breakdowns/slam-mirrorbot Slam Mirror Bot is a Python-based multipurpose Telegram bot that mirrors files from torrents, direct links, Mega, and Telegram to Google Dr… | 10 | 1255 | abandoned |
| Zeal-L/BiliBili-Manga-Downloader A GUI-based Bilibili Manga downloader built with Python and PySide6 that supports keyword search, QR-code login, multi-threaded downloads, … | 34 | 1248 | abandoned |
| vaycore/OneScan OneScan is a BurpSuite extension written in Java for recursive directory scanning, helping discover hidden vulnerabilities in deeper direct… | 10 | 1247 | abandoned |
| n0tr00t/Sreg Sreg is a Python CLI OSINT tool that checks whether an email, phone number, or username has been used to register accounts across many Chin… | 32 | 1246 | abandoned |
| prose-intelligence-ltd/Telepathy-Community Telepathy-Community is a Python command-line OSINT toolkit for collecting and analyzing public Telegram chats, including archiving messages… | 72 | 1241 | abandoned |
| syrusakbary/gdom GDOM is a Python CLI tool that lets you scrape and traverse web page DOM using GraphQL queries, built on the Graphene framework. You write … | 32 | 1239 | abandoned |
| 0xlousie/OSIF OSIF is a Python command-line tool that gathers sensitive information from Facebook accounts, such as residence, date of birth, occupation,… | 32 | 1236 | abandoned |
| the-robot/sqliv SQLiv is a Python command-line tool that scans websites for SQL injection vulnerabilities. It supports dork-based scanning via search engin… | 10 | 1232 | abandoned |
| chenjiandongx/mzitu A Python web crawler that downloads full photo galleries from mzitu.com, collecting over 3,000 image sets. It also performs word-frequency … | 32 | 1230 | abandoned |
| eprendre/tingshu An Android app for streaming audiobooks from multiple free online audiobook sites, written in Kotlin. It supports user-written custom sourc… | 56 | 1229 | abandoned |
| FeezyHendrix/Insta-mass-account-creator A Python CLI tool that automatically mass-creates Instagram accounts, generating disposable email addresses, fetching OTPs, and populating … | 10 | 1203 | abandoned |
| mintapi/mintapi An unofficial Python library and CLI that screen-scrapes Mint.com using Selenium/WebDriver to programmatically retrieve account, transactio… | 32 | 1199 | abandoned |
| foolcage/fooltrader fooltrader is a Python quantitative analysis and trading framework that crawls, cleans, and structures market data (stocks, futures, forex,… | 23 | 1198 | abandoned |
| jito-labs/mev-bot A TypeScript bot that performs backrun arbitrage on Solana by monitoring the Jito mempool for large trades and executing circular arbitrage… | 36 | 1196 | abandoned |
| ClaudiuGeorgiu/PlaystoreDownloader A Python command line tool (with an optional web interface) that downloads Android APK files directly from the Google Play Store by package… | 10 | 1193 | abandoned |
| k1995/BaiduyunSpider A distributed crawler and search engine for Baidu Netdisk (Baidu Cloud) shared links, built with Scrapy and Scrapy-Redis, with a React-base… | 23 | 1182 | abandoned |
| 4x99/code6 Code6 (码小六) is a self-hosted web application that monitors GitHub for leaked code and secrets, scanning periodically and alerting via email… | 23 | 1181 | abandoned |
| xiaoxu193/PyTeaser PyTeaser is a Python library that summarizes news articles by ranking sentences for relevance to the title, keywords, position, and length,… | 75 | 1170 | abandoned |
| natewong1313/bird-bot Bird Bot is a Python desktop auto-checkout bot with a PyQt5 interface that monitors Walmart and Best Buy for Nintendo Switch console restoc… | 10 | 1169 | abandoned |
| xiaoguyu/wechatDownload A desktop application built with Electron, TypeScript, and Vue3 for downloading WeChat Official Account (公众号) articles. It captures require… | 10 | 1168 | abandoned |
| shadowmoose/RedditDownloader Reddit Media Downloader is a Python CLI tool that scans Reddit posts, comments, saved lists, and subreddits to download linked media locall… | 10 | 1156 | abandoned |
| jamietre/CsQuery CsQuery is a C# jQuery port for .NET 4 that provides a complete CSS2/CSS3 selector engine, a standards-compliant HTML parser, and jQuery-st… | 32 | 1148 | abandoned |
| xillwillx/skiptracer Skiptracer is a Python-based OSINT web scraping framework that aggregates paid and free lookup services to enumerate target information suc… | 23 | 1148 | abandoned |
| Jinnrry/getAwayBSG A Go CLI web crawler that scrapes job listings from Zhilian Zhaopin and housing (rental and second-hand) data from Lianjia across Chinese c… | 10 | 1144 | abandoned |
| ring04h/weakfilescan A Python-based multi-threaded sensitive information leakage detection tool that crawls a target site, dynamically builds dictionary rules f… | 32 | 1140 | abandoned |
| vesche/scanless A Python CLI utility and library that scrapes online port scanning websites to perform port scans on a target IP or domain on your behalf. … | 10 | 1119 | abandoned |
| joshhighet/ransomwatch A transparent ransomware claim tracker that monitors ransomware groups' leak sites on the dark web and records their victim claims. It aggr… | 10 | 1114 | abandoned |
| fzaninotto/screenshot-as-a-service A self-hosted Node.js web service that renders PNG screenshots of websites using PhantomJS behind an Express HTTP API. It supports custom v… | 10 | 1096 | abandoned |
| AllanWang/Frost-for-Facebook Frost is an open-source third-party Facebook client for Android built as a web wrapper with extensive theming, multi-account support, and n… | 10 | 1095 | abandoned |
| bytebuff/JSpider JSpider is a collection of JavaScript decryption files for website-encrypted request parameters, shared weekly alongside Python snippets th… | 32 | 1091 | abandoned |
| TobiaszCudnik/phpquery phpQuery is a server-side, chainable, CSS3-selector-driven DOM API for PHP5 modeled on the jQuery JavaScript library. It also ships a comma… | 10 | 1065 | abandoned |
| VikParuchuri/apartment-finder A Python Slack bot that scrapes Craigslist for real-time apartment listings matching configurable criteria (price, neighborhoods, transit p… | 10 | 1058 | abandoned |
| iBreaker/bjguahao A Python-based helper script for booking appointments on the Beijing unified hospital registration platform (北京市预约挂号统一平台). The source code … | 32 | 1056 | abandoned |
| loklak/loklak_jlib_api A Java client library for the Loklak server, which scrapes and provides open access to social media data (primarily Twitter) without requir… | 32 | 1055 | abandoned |
| cloudroc/baidu-nolimit A tiny Chrome extension that bypasses Baidu Netdisk's large-file download restrictions by spoofing the navigator.platform property so the w… | 32 | 1041 | abandoned |
| 7sDream/zhihu-py3 An unofficial Python 3 API library for Zhihu, the Chinese Q&A site, letting users build objects from Zhihu URLs to fetch questions, answers… | 10 | 1033 | abandoned |
| ahertel/Amazon-Fresh-Whole-Foods-delivery-slot-finder A Mac-only AppleScript tool that repeatedly checks Amazon's Whole Foods delivery and Amazon Fresh pages to find available delivery slots an… | 32 | 1020 | abandoned |
| SpiderClub/smart_login A Python collection of simulated login implementations for major Chinese websites (Weibo, Zhihu, QQ Zone, JD, Baidu, etc.), using either di… | 32 | 1008 | abandoned |
| Planetary-Computers/autotab-starter autotab-starter is a Python toolkit for building AI-powered browser agents by recording point-and-click demonstrations in Chrome and genera… | 28 | 1007 | abandoned |
| owenthereal/hacker-menu Hacker Menu is a macOS menu bar desktop application that delivers top Hacker News stories directly to your desktop. It is built with Electr… | 23 | 1000 | abandoned |
| Netflix-Skunkworks/sketchy Sketchy is a task-based API for taking screenshots and scraping text from websites, built on PhantomJS and Celery with optional token auth … | 32 | 998 | abandoned |
| Hermes Agent Hermes Agent is an open‑source autonomous AI agent that runs on the command line or as a native desktop app. It continuously learns from in… | 86 | 254714 | active |
| yt-dlp yt-dlp is a feature-rich command-line audio/video downloader supporting thousands of websites, forked from youtube-dl. It offers extensive … | 95 | 195314 | active |
| Playwright Playwright is an open‑source framework for web automation and end‑to‑end testing. It provides a single API to control Chromium, Firefox, an… | 100 | 98243 | active |
| Sherlock Sherlock is a Python CLI tool that hunts down social media accounts by username across 400+ social networks. It is widely used for OSINT re… | 85 | 92935 | active |
| OpenHands OpenHands is an open‑source platform that provides a web‑based GUI, CLI and SDK for building, running and orchestrating AI coding agents. I… | 88 | 90527 | active |
| Odysseus AI Workspace Odysseus is a self‑hosted AI workspace that combines chat, agents, document editing, email, notes, and calendar features with local model s… | 57 | 87526 | active |
| Odysseus AI Workspace Odysseus is a self-hosted, privacy-first AI workspace that combines chat, autonomous agents, tools/MCP integration, model serving, and deep… | 57 | 87526 | active |
| AnythingLLM AnythingLLM is an all-in-one, local-first AI application for chatting with your documents, running AI agents, and building workflows entire… | 97 | 67412 | active |
| sansan0/TrendRadar TrendRadar is a self-hostable AI-powered public opinion and trending-news monitor that aggregates hot topics across multiple platforms and … | 62 | 62570 | active |
| Alishahryar1/free-claude-code Free Claude Code is an open‑source Python CLI and library that lets you access multiple LLM coding agents (Claude, Codex, Pi, OpenCode, etc… | 79 | 61286 | active |
| MadsLorentzen/ai-job-search An AI-powered job application framework built on Claude Code that runs locally: it evaluates job postings, tailors CVs, writes cover letter… | 83 | 60836 | active |
| Joplin Joplin is an open-source, privacy-focused note-taking and to-do application that stores notes in Markdown and supports notebooks, tags, and… | 99 | 56805 | stable |