Open-source web-scraping software
2000 open-source libraries, tools and applications tagged web-scraping, ranked by how closely web-scraping is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
2000 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| jsvine/waybackpack Waybackpack is a Python command-line tool that downloads the entire Wayback Machine archive for a given URL, saving every archived snapshot… | 27 | 3235 | active |
| Bionus/imgbrd-grabber Grabber is a highly customizable imageboard/booru browser and mass downloader that can fetch thousands of images from multiple booru source… | 84 | 3224 | active |
| lackeyjb/playwright-skill An Agent Skill and Claude Code plugin that lets coding agents write and execute Playwright automation scripts on the fly, from simple page … | 80 | 3221 | active |
| devanshbatham/ParamSpider ParamSpider is a Python CLI tool that mines URLs for a domain or list of domains from web archives (Wayback Machine), filtering out uninter… | 51 | 3190 | active |
| s0md3v/Photon Photon is a fast Python-based web crawler designed for OSINT (open-source intelligence) tasks. It extracts URLs, emails, social media accou… | 65 | 13243 | maintenance |
| nilbuild/githunt GitHunt is a React web application and Chrome extension that lets users explore the most starred GitHub projects by week, with filtering by… | 56 | 3189 | active |
| pingc0y/URLFinder URLFinder is a fast, easy-to-use Go CLI tool that extracts JS files, URLs, and sensitive information from web pages, including hidden unaut… | 73 | 3181 | active |
| axcore/tartube Tartube is a free, open-source GUI front-end for youtube-dl, yt-dlp and other compatible video downloaders, written in Python 3 / Gtk 3. It… | 82 | 3166 | active |
| feder-cr/invisible_playwright A Python library that provides an antidetect, stealth-patched Firefox build for Playwright, with fingerprints set at the C++ engine level a… | 78 | 3160 | active |
| keon/browser-control A small, fast Rust CLI that drives a real browser over the Chrome DevTools Protocol, designed for coding agents and shell pipelines. It exp… | 90 | 3151 | active |
| Virtual-Browser/VirtualBrowser VirtualBrowser is a free, open-source anti-fingerprint browser built on Chromium that lets users create and manage multiple isolated browse… | 88 | 3145 | active |
| thp/urlwatch urlwatch is a Python CLI tool that monitors webpages for changes and notifies you via e-mail, terminal, or third-party services like Telegr… | 70 | 3142 | active |
| sabnzbd/sabnzbd SABnzbd is an open-source binary newsreader written in Python that automates Usenet downloads from NZB files, handling downloading, verific… | 96 | 3121 | active |
| adryfish/fingerprint-chromium A fingerprint browser built on Ungoogled Chromium that lets users spoof or control browser fingerprint characteristics to avoid detection. … | 83 | 3110 | active |
| CharlesPikachu/videodl A lightweight video downloader written in pure Python that parses and downloads videos from dozens of streaming platforms (Douyin, Bilibili… | 81 | 3105 | active |
| sergiotapia/magnetissimo Magnetissimo is a self-hosted Elixir/Phoenix web application that crawls popular torrent sites and indexes magnet links into a local Postgr… | 99 | 3100 | active |
| Scrapyd Scrapyd is a service daemon for deploying and running Scrapy spiders. It lets you upload Scrapy projects and control spiders via a JSON HTT… | 66 | 3098 | stable |
| antimatter15/splat A WebGL-based real-time viewer for 3D Gaussian Splatting scenes, rendering photorealistic navigable 3D environments from photo-derived spla… | 48 | 3071 | active |
| lxml/lxml lxml is a fast, feature-rich Python library for processing XML and HTML, built on the C libraries libxml2 and libxslt. It provides an Eleme… | 97 | 3067 | stable |
| symfony/panther Symfony Panther is a PHP library for browser testing and web scraping that drives real browsers (Chrome, Firefox) via the W3C WebDriver pro… | 66 | 3067 | active |
| spacecowboy/Feeder Feeder is an open-source RSS/Atom/JSON Feed reader app for Android, built with Kotlin and Jetpack Compose. It runs entirely locally with no… | 97 | 3053 | active |
| web-scrobbler/web-scrobbler Web Scrobbler is a browser extension that scrobbles music playback history from streaming sites to services like Last.fm, Libre.fm, ListenB… | 92 | 3049 | active |
| Alfredredbird/tookie-osint Tookie-OSINT is an open-source Python OSINT tool that finds social media accounts and gathers information based on user inputs. It offers a… | 81 | 3046 | active |
| Qianlitp/crawlergo crawlergo is a Go-based browser crawler that uses headless Chrome to discover URLs for web vulnerability scanners. It renders pages, fills … | 24 | 3039 | active |
| oxylabs/perplexity-scraper A repository of code examples and documentation for Oxylabs' Perplexity Scraper API, which sends prompts to Perplexity and returns AI-gener… | 61 | 3029 | active |
| dw-dengwei/daily-arXiv-ai-enhanced A tool that automatically crawls daily arXiv papers and summarizes them using LLMs like DeepSeek, publishing the results as a browsable sit… | 62 | 3027 | active |
| TheBlewish/Automated-AI-Web-Researcher-Ollama A Python application that turns a locally-run Ollama LLM into an automated web research assistant. Given a single query, it generates focus… | 63 | 3017 | active |
| deedy5/ddgs DDGS (Dux Distributed Global Search) is a Python metasearch library that aggregates text, image, video, news, and book results from multipl… | 89 | 3013 | active |
| vitalets/github-trending-repos A GitHub-based service that tracks trending repositories per programming language and delivers updates via native GitHub notifications. A s… | 52 | 3013 | active |
| MALSync/MALSync MAL-Sync is a browser extension and userscript that integrates MyAnimeList, AniList, Kitsu, Simkl, Shikimori, and MangaBaka into hundreds o… | 93 | 3012 | active |
| symfony/browser-kit Symfony BrowserKit is a PHP component that simulates the behavior of a web browser, enabling programmatic requests, link clicks, and form s… | 93 | 3008 | stable |
| thewhiteh4t/FinalRecon FinalRecon is an all-in-one automatic web reconnaissance tool written in Python that provides a fast overview of a web target. It bundles h… | 67 | 3006 | active |
| AutoCLI AutoCLI is a blazing-fast, memory-safe command-line tool written in Rust that fetches information from 55+ websites (Twitter/X, Reddit, You… | 60 | 2992 | active |
| Video_Spider A PHP-based web service that parses short-video links from platforms like Douyin, Kuaishou, Weibo, and Pipixia to return watermark-free vid… | 53 | 2991 | active |
| striver-ing/wechat-spider An open-source WeChat crawler that scrapes articles, reading counts, likes, and comments from WeChat official accounts using a man-in-the-m… | 73 | 2973 | active |
| Tyrrrz/DiscordChatExporter DiscordChatExporter is an application that exports Discord message history from direct messages, group messages, and server channels to fil… | 92 | 12116 | maintenance |
| AcadHomepage AcadHomepage is a Jekyll-based template for building a modern, responsive academic personal homepage hosted on GitHub Pages. It automatical… | 74 | 2957 | active |
| rust-headless-chrome/rust-headless-chrome A Rust library providing a high-level API to control headless Chrome or Chromium via the DevTools Protocol, serving as the Rust equivalent … | 79 | 2952 | active |
| public-clis/twitter-cli A terminal-first CLI for Twitter/X that lets users read timelines, bookmarks, search results, and user profiles without official API keys, … | 48 | 2947 | active |
| Luoyacheng/legado-E 阅读Sigma (Reading Sigma) is an open-source Android e-book reader forked from Legado, offering customizable book sources, RSS subscriptions, … | 69 | 2939 | active |
| GodsScion/Auto_job_applier_linkedIn A Python/Selenium desktop application that automates LinkedIn Easy Apply job applications, finding relevant jobs, filling application quest… | 93 | 2931 | active |
| langchain-ai/social-media-agent A TypeScript AI agent built on LangGraph that takes a URL, scrapes its content, and generates Twitter and LinkedIn posts using LLMs. It inc… | 67 | 2920 | active |
| LittleSurvival/copymanga-copy20 A collection of Chinese manga source extensions for Tachiyomi/Mihon-compatible comic reader apps, supporting Copymanga, vomic, Baozimanhua,… | 87 | 2918 | active |
| ChanceYu/front-end-rss An automated RSS aggregator that collects the latest front-end technology articles from popular newsletters and blogs, categorizes them, an… | 77 | 2901 | active |
| vladkens/twscrape twscrape is an async Python library and CLI for scraping X/Twitter via its Search and GraphQL endpoints using a pool of your own accounts. … | 95 | 2882 | active |
| zhizhuodemao/js-reverse-mcp An MCP server that gives AI coding agents (Claude, Cursor, Copilot) tools for JavaScript reverse engineering in a headed Chrome browser, in… | 79 | 2881 | active |
| cdhigh/KindleEar KindleEar is a self-hostable Python web application that aggregates RSS/ATOM/JSON feeds and web content (including Calibre recipes) into ep… | 73 | 2875 | active |
| littledivy/mimic mimic is a Python tool that captures traffic from mobile or web apps via mitmproxy, extracts authentication material, and uses AI to genera… | 52 | 2870 | active |
| goniszewski/grimoire Grimoire is a local-first, self-hosted bookmark manager for saving, extracting, and searching web content, with optional AI-powered summari… | 95 | 2866 | active |
| ScrapeCreators/social-media-research-skills A collection of AI agent skills for social media research built on the ScrapeCreators scraping API. The skills give agents complete workflo… | 55 | 2863 | active |
| digininja/CeWL CeWL is a Ruby command-line tool that spiders a target website to a specified depth and collects unique words into a custom wordlist for us… | 59 | 2862 | stable |
| zzzprojects/html-agility-pack Html Agility Pack (HAP) is a free, open-source HTML parser written in C# that builds a read/write DOM and supports XPath and XSLT queries. … | 89 | 2850 | active |
| youniaogu/MangaReader A cross-platform manga reading app built with React Native for Android and iOS, with tablet support. It uses a plugin-based design to aggre… | 48 | 2833 | active |
| spatie/crawler A PHP library by Spatie for crawling links on websites, built on Guzzle promises for concurrent requests. It can execute JavaScript via Chr… | 91 | 2831 | active |
| chrieke/prettymapp Prettymapp is a Python package and Streamlit webapp that generates stylized, artistic maps from OpenStreetMap data. It is a speed-focused r… | 76 | 2827 | active |
| Geniusay/ChopperBot ChopperBot is a fully automated AI bot that monitors popular live streams across platforms like Douyu, Huya, Bilibili, Douyin, and Twitch, … | 42 | 2826 | active |
| besuper/TwitchNoSub A browser extension for Chrome, Chromium-based browsers, and Firefox that lets users watch subscriber-only VODs (past broadcasts) on Twitch… | 73 | 2813 | active |
| meeb/tubesync TubeSync is a self-hosted PVR (personal video recorder) for YouTube that syncs channels and playlists to local directories, wrapping yt-dlp… | 99 | 2807 | active |
| rusq/slackdump Slackdump is a Go CLI tool that archives private and public Slack messages, threads, files, users, channels, and emojis locally without req… | 97 | 2807 | active |
| Spider Flow Spider-Flow is a self-hosted Java-based web crawler platform that lets users define scraping workflows visually as flowcharts without writi… | 23 | 11351 | maintenance |
| geziyor/geziyor Geziyor is a fast web crawling and scraping framework for Go, supporting JavaScript rendering via Chrome, caching, proxy management, and au… | 69 | 2782 | active |
| xnl-h4ck3r/waymore waymore is a Python CLI tool that retrieves URLs from multiple web archive and intelligence sources (Wayback Machine, Common Crawl, Alien V… | 84 | 2762 | active |
| chaolucky18/xuexitongScript A Tampermonkey userscript (also runnable via browser console) that automates course video playback on the Xuexitong (Chaoxing) learning pla… | 70 | 2759 | active |
| cobrateam/splinter Splinter is a Python library providing a simple, consistent API for browser automation and web application acceptance testing. It abstracts… | 36 | 2750 | active |
| Spider Spider is a concurrency-first web crawler and scraper written in Rust that streams pages as they arrive, renders JavaScript only when neede… | 83 | 2747 | active |
| Dineshkarthik/telegram_media_downloader A Python tool that downloads media files (audio, documents, photos, videos, voice notes) from Telegram chats, channels, and conversations, … | 75 | 2742 | active |
| microlinkhq/metascraper Metascraper is a Node.js library that extracts unified metadata from any URL by combining Open Graph, JSON-LD, Microdata, RDFa, Twitter Car… | 92 | 2738 | active |
| geekgeekrun/geekgeekrun GeekGeekRun is a free, open-source desktop application (built on Electron, Vue, Puppeteer, SQLite/TypeORM) that automates job hunting on th… | 73 | 2732 | active |
| prinsss/twitter-web-exporter A UserScript (for Tampermonkey/Violentmonkey) that exports tweets, bookmarks, lists, followers, search results, and media from the Twitter/… | 86 | 2721 | active |
| zubair-trabzada/ai-marketing-claude A suite of 15 marketing skills for Claude Code that runs parallel subagents to audit websites, generate copy, email sequences, ad creatives… | 43 | 2714 | active |
| Johnserf-Seed/f2 F2 is an asynchronous Python library and CLI tool for downloading videos and fetching API data from multiple platforms including Douyin, Ti… | 67 | 2704 | active |
| TermuxHackz/X-osint X-osint is an open-source Python-based OSINT framework for gathering information about phone numbers, email addresses, IP addresses, VINs, … | 69 | 2701 | active |
| Nandaka/PixivUtil2 A Python command-line tool for bulk downloading images from Pixiv and Pixiv FANBOX, with support for downloading by member, tag, bookmark, … | 69 | 2696 | active |
| MrsEWE44/musicDownload A Python GUI application for searching and downloading music from major Chinese music platforms (Kugou, Kuwo, QQ Music, NetEase Cloud Music… | 63 | 2694 | active |
| Open-Web-Analytics/Open-Web-Analytics Open Web Analytics (OWA) is an open source, self-hosted web analytics server and JavaScript tracking client, serving as an alternative to G… | 97 | 2690 | active |
| jae-jae/QueryList QueryList is a progressive PHP web scraping framework built on phpQuery that provides jQuery-like CSS3 DOM selectors and manipulation APIs … | 72 | 2690 | active |
| chrome-php/chrome A PHP library for controlling headless Chrome/Chromium browsers via the DevTools protocol, supporting both synchronous and asynchronous usa… | 83 | 2681 | active |
| jackwener/xiaohongshu-cli A Python CLI for Xiaohongshu (小红书/Red) that lets users search, read, and interact with notes, comments, feeds, and profiles via a reverse-e… | 45 | 2679 | active |
| ShareDropio/sharedrop ShareDrop is a web application inspired by Apple AirDrop that transfers files directly between devices using WebRTC peer-to-peer connection… | 32 | 10752 | maintenance |
| pt-plugins/PT-depiler PT-depiler is a Manifest v3 browser extension that improves efficiency when using private tracker (PT) sites, succeeding PT-Plugin-Plus. It… | 100 | 2670 | active |
| lncrawl/lightnovel-crawler Lightnovel Crawler is a Python tool that downloads web novels from 300+ supported sources and converts them into e-books such as EPUB, MOBI… | 96 | 2669 | active |
| Bright Data MCP A Model Context Protocol (MCP) server by Bright Data that gives AI agents and LLMs real-time access to public web data through 69 tools cov… | 86 | 2663 | active |
| nobiyou/wx_channel A Windows desktop tool that injects download buttons into WeChat Channels (视频号) pages, enabling one-click and batch downloading of videos i… | 81 | 2662 | active |
| agentrhq/webcmd Webcmd is a self-learning browser infrastructure CLI for AI agents that compiles knowledge of websites into deterministic commands, cutting… | 78 | 2639 | active |
| botswin/BotBrowser BotBrowser is a privacy-focused browser core (Chromium-based) that unifies and controls browser fingerprint signals across platforms, integ… | 85 | 2632 | active |
| amtoaer/bili-sync bili-sync is a Bilibili video synchronization tool written in Rust and Tokio, designed for NAS users. It automatically downloads favorites,… | 84 | 2623 | active |
| spatie/laravel-sitemap A Laravel package by Spatie that generates XML sitemaps, either by crawling an entire site automatically or by adding URLs manually (includ… | 89 | 2621 | stable |
| apify/fingerprint-suite A modular TypeScript toolkit by Apify for generating realistic browser fingerprints and HTTP headers and injecting them into Playwright or … | 95 | 2619 | active |
| raviqqe/muffet Muffet is a fast website link checker written in Go that recursively scrapes and inspects all pages of a website for broken links. It suppo… | 95 | 2616 | active |
| Serene-Arc/bulk-downloader-for-reddit A Python command-line tool (bdfr) that bulk-downloads and archives Reddit submissions and their media from subreddits, multireddits, users,… | 53 | 2613 | active |
| sarperavci/CloudflareBypassForScraping A Python library that bypasses Cloudflare's anti-bot verification for web scraping, supporting cookie generation and request mirroring for … | 68 | 2603 | active |
| pi-skills A collection of skills (SKILL.md-based tool packages) for the pi coding agent, also compatible with Claude Code, Codex CLI, Amp, and Droid.… | 52 | 2592 | active |
| simonw/shot-scraper shot-scraper is a Python CLI utility built on Playwright for taking automated screenshots of websites, recording video demos, and scraping … | 92 | 2582 | active |
| miantiao-me/hacker-podcast An AI-powered Chinese podcast application that automatically scrapes daily Hacker News top stories, generates Chinese summaries and scripts… | 63 | 2581 | active |
| liu673cn/bug An open-source TVBox shell application (empty shell) for Android TV and mobile devices that plays video streams from user-supplied configur… | 32 | 10324 | maintenance |
| krau/SaveAny-Bot A self-hosted Telegram bot written in Go that saves Telegram files (documents, videos, photos, stickers, Telegraph pages) to various storag… | 85 | 2571 | active |
| owntone/owntone-server OwnTone is an open-source audio media server written in C that streams local files, Spotify, and internet radio to AirPlay 1/2 (multiroom),… | 89 | 2560 | active |
| kpcyrd/sn0int sn0int is a semi-automatic OSINT framework and package manager written in Rust that enumerates attack surface by processing public informat… | 66 | 2553 | active |
| get_iplayer A Perl-based command-line tool that indexes and downloads TV and radio programmes from BBC iPlayer and BBC Sounds. It supports regex search… | 28 | 2552 | active |
| zotify-dev/zotify Zotify is a command-line tool for downloading music tracks, albums, playlists, and podcasts directly from streaming sources at up to 320kbp… | 32 | 2548 | active |