Open-source web-scraping software
2000 open-source libraries, tools and applications tagged web-scraping, ranked by how closely web-scraping is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
2000 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| dilame/instagram-private-api A NodeJS/TypeScript SDK providing a client for Instagram's private (undocumented) API, enabling full programmatic access to feeds, direct m… | 23 | 6480 | maintenance |
| jldbc/pybaseball pybaseball is a Python package that scrapes and retrieves current and historical baseball statistics from sources like MLB Statcast (Baseba… | 46 | 1735 | active |
| claffin/cloudproxy CloudProxy is a self-hosted Python tool that provisions and manages proxy servers across multiple cloud providers, rotating IPs to improve … | 77 | 1727 | active |
| qinlili23333/ctfileGet A web-based resolver that obtains one-time direct download URLs for files hosted on Chengtong Network Disk (ctfile/城通网盘) using its official… | 52 | 1722 | active |
| lanyeeee/jmcomic-downloader A multi-threaded GUI downloader for the 18comic.vip (jmcomic) manga site, built with Tauri (Rust backend, Vue frontend). It supports search… | 84 | 1721 | active |
| openkursar/hello-halo Halo is an open-source desktop AI workstation that wraps frontier coding agents like Claude Code and Codex in a visual GUI, with a pluggabl… | 81 | 1719 | active |
| ka-pi-ba-la/AIbijia AIbijia is a price-comparison website and community resource that aggregates AI subscription and token prices across multiple platforms (Ch… | 54 | 1713 | active |
| Jules-WinnfieldX/CyberDropDownloader A Python-based bulk downloader that scrapes and downloads files from Cyberdrop.me and dozens of other file hosts and image galleries. It is… | 10 | 1712 | active |
| wu529778790/panhub.shenzjd.com PanHub is a self-hostable netdisk search aggregator that combines results from Quark, Aliyun Drive, Baidu Netdisk, 115, Thunder and 80+ Tel… | 62 | 1708 | active |
| Python3Spiders/WeiboSuperSpider A Weibo (Chinese microblog) scraping toolbox in Python covering users, topics, and comments, with extras like image downloading, sentiment … | 71 | 1706 | active |
| aisingapore/TagUI TagUI is a free, open-source robotic process automation (RPA) tool from AI Singapore that lets users write simple text flows to automate re… | 66 | 6336 | maintenance |
| ion-design/ditto.site ditto.site is a hosted capture-to-code service that turns a public URL into a self-contained TypeScript app, emitting a deterministic Next.… | 58 | 1704 | active |
| utkusen/urlhunter urlhunter is a Go-based recon CLI tool that searches URLs exposed via shortener services like bit.ly and goo.gl. It downloads daily URLTeam… | 23 | 1701 | active |
| rushter/selectolax Selectolax is a fast Python HTML5 parser library written in Cython, binding to the Modest and Lexbor parsing engines. It provides CSS selec… | 96 | 1681 | active |
| Ge0rg3/requests-ip-rotator A Python library that mounts AWS API Gateway as a proxy onto requests sessions, rotating source IPs on every request using AWS's large IP p… | 84 | 1680 | active |
| vladko312/SSTImap SSTImap is a Python-based penetration testing tool that automatically detects and exploits Server-Side Template Injection (SSTI) and code i… | 82 | 1679 | active |
| jarun/googler googler is a Python command-line tool for performing Google web, news, and video searches directly from the terminal. It displays titles, U… | 10 | 6208 | maintenance |
| Benexl/yt-x A POSIX-compliant shell script that lets you browse YouTube and other yt-dlp-supported sites from the terminal using fzf or from an app lau… | 89 | 1668 | active |
| GramAddict/bot GramAddict is a free, open-source Instagram bot that automates liking, following, commenting, and messaging on any Android device or emulat… | 23 | 1664 | active |
| nicobailon/pi-web-access A TypeScript extension for the Pi coding agent that adds web search, URL content extraction, GitHub repo cloning, PDF extraction, and YouTu… | 84 | 1663 | active |
| srx-2000/spider_collection A collection of Python web crawler scripts targeting sites like Bilibili, Zhihu, Weibo, NetEase Music, GitHub, and Anjuke, built with reque… | 32 | 1657 | active |
| ScopeSentry ScopeSentry is a self-hosted attack surface and asset mapping platform that combines subdomain enumeration, port scanning, fingerprinting, … | 83 | 1653 | active |
| Sparticuz/chromium A TypeScript library that packages a serverless-optimized Chromium binary (Brotli-compressed) with decompression code and predefined launch… | 96 | 1651 | active |
| itteco/iframely Iframely is a self-hosted oEmbed proxy and URL metadata service that takes any URL and returns rich media embed codes and page metadata. It… | 88 | 1651 | active |
| oxylabs/ai-map-py AI-Map is a Python SDK client for Oxylabs AI Studio's AI-powered website mapping service, which discovers and extracts relevant URLs from a… | 10 | 1651 | active |
| itsToggle/plex_debrid A Python application that monitors Plex, Trakt, and Overseerr watchlists and automatically fetches requested movies and shows from torrent … | 10 | 1647 | active |
| vigorX777/ai-daily-digest A zero-dependency TypeScript CLI tool (runnable as an OpenCode skill) that fetches the latest posts from 90 top tech blogs recommended by A… | 42 | 1643 | active |
| Altimis/Scweet Scweet is a Python library and CLI for scraping tweets, profile timelines, followers, following lists, and user profiles from Twitter/X wit… | 98 | 1640 | active |
| Pickle-Pixel/ApplyPilot ApplyPilot is an open-source AI agent that autonomously applies to jobs on your behalf across any job site or application form. It runs a 6… | 54 | 1638 | active |
| qsniyg/maxurl Image Max URL is a userscript and browser extension that finds larger or original versions of images and videos by rewriting URL patterns, … | 75 | 1630 | active |
| michenriksen/aquatone Aquatone is a Go CLI tool for visual inspection of websites across many hosts, taking screenshots via headless Chrome/Chromium and generati… | 10 | 5962 | maintenance |
| hartator/wayback-machine-downloader A Ruby command-line tool that downloads an entire website from the Internet Archive Wayback Machine, restoring original files and directory… | 23 | 5958 | maintenance |
| ArchiveTeam/grab-site grab-site is a preconfigured web crawler for archiving websites, producing WARC files via a fork of wpull. It includes a dashboard for moni… | 39 | 1614 | active |
| m8sec/CrossLinked CrossLinked is a Python CLI tool that enumerates LinkedIn employee names for an organization by scraping search engine results, without nee… | 23 | 1611 | active |
| Mink Mink is a PHP library providing an abstraction layer over web browser emulators and drivers (like Goutte, Selenium) for controlling browser… | 68 | 1608 | active |
| actionbook/actionbook Actionbook is an AI agent that lives in a Chrome side panel and operates websites on your behalf, reading data behind logins and paywalls a… | 71 | 1607 | active |
| matthewmueller/x-ray x-ray is a Node.js web scraping library that lets you define flexible schemas to structure data from any website using jQuery-like selector… | 64 | 5909 | maintenance |
| ttttmr/Wechat2RSS Wechat2RSS is a service and self-hostable tool that converts WeChat official account (公众号) articles into RSS feeds, aiming for updates with… | 71 | 1606 | active |
| rafatosta/zapzap ZapZap is an unofficial WhatsApp Web desktop client built with Python, PyQt6, and QtWebEngine that wraps web.whatsapp.com in a native deskt… | 93 | 1600 | active |
| myfanhua/turb-gpt-free-register A Python tool that bulk-registers ChatGPT/OpenAI accounts using pure-protocol requests or anti-fingerprint browser automation (RoxyBrowser,… | 58 | 1600 | active |
| Ademking/MD-This-Page A browser extension for Chrome and Firefox that converts any webpage into clean, readable Markdown with one click, using Mozilla's Readabil… | 73 | 1598 | active |
| Hyperbrowser HyperAgent is a TypeScript library and CLI that adds LLM-powered natural language commands to Playwright for browser automation. It support… | 47 | 1598 | active |
| LeyckerS/moondownloader Moon Downloader is a bulk file downloader for the datanodes.to and fuckingfast.co file hosts, extracting direct links either via a real Chr… | 88 | 1597 | active |
| xnl-h4ck3r/xnLinkFinder xnLinkFinder is a Python CLI tool that discovers endpoints, potential parameters, target-specific wordlists, and secrets for a given target… | 71 | 1594 | active |
| s0md3v/uro uro is a Python CLI tool that declutters URL lists for crawling and security testing without making any HTTP requests. It removes duplicate… | 26 | 1594 | stable |
| m3n0sd0n4ld/GooFuzz GooFuzz is a Bash-based CLI tool that performs fuzzing-style reconnaissance using advanced Google searches (Google Dorking) via the Google … | 50 | 1591 | active |
| Xatta-Trone/medium-parser-extension A browser extension for Chrome, Edge, and Firefox that lets users read member-only articles on medium.com and Medium-based sites (e.g. Towa… | 56 | 1590 | active |
| AlisamTechnology/ATSCAN ATSCAN is a Perl-based command-line scanner for mass dork searching and vulnerability exploitation. It combines search engine dorking with … | 23 | 1589 | active |
| webrecorder/archiveweb.page ArchiveWeb.page is a high-fidelity web archiving tool that runs as a Chrome/Chromium browser extension and as a standalone Electron app. It… | 94 | 1587 | active |
| jasperan/whatsapp-osint A Python CLI tool that uses Selenium to track when WhatsApp contacts go online/offline, logging presence sessions to SQLite. It exports dat… | 77 | 1580 | active |
| Postlight Parser Postlight Parser is a JavaScript library that extracts meaningful content—article text, titles, authors, dates, lead images, and excerpts—f… | 23 | 5791 | maintenance |
| tafia/quick-xml quick-xml is a high-performance XML pull reader and writer library for Rust, featuring near zero-copy parsing with Cow types and buffer reu… | 89 | 1570 | stable |
| Leon406/SubCrawler A Kotlin-based tool that automatically crawls and health-checks (via Google ping) free public proxy nodes for protocols like V2Ray, Shadows… | 77 | 1570 | active |
| OWASP/QRLJacking QRLJacking is an OWASP project documenting and exploiting the Quick Response Code Login Jacking attack vector, which hijacks user sessions … | 44 | 1567 | active |
| cross-seed/cross-seed cross-seed is a Node.js application that automatically finds and downloads torrents matching your existing torrent library across multiple … | 89 | 1566 | active |
| oxylabs/browser-agent-py A Python SDK for Oxylabs AI Studio's Browser Agent, a cloud service that automates real-user browsing tasks (clicking, typing, scrolling, s… | 10 | 1565 | active |
| skernelx/tavily-key-generator A Python toolkit that automates signup flows for Tavily, Firecrawl, and Exa using real browser automation (Playwright/Camoufox), Turnstile … | 44 | 1560 | active |
| Ulixee Hero Hero is a headless web browser built specifically for web scraping, powered by Chrome and controlled from NodeJS with a fully compliant DOM… | 59 | 1557 | active |
| FinanceData/FinanceDataReader FinanceDataReader is a Python library and CLI tool for reading financial data such as stock listings, stock prices, indexes, exchange rates… | 66 | 1555 | active |
| eatmoreduck/boss-zhipin-scraper A Python CLI scraper for BOSS Zhipin (zhipin.com) that connects to a locally logged-in Chrome via the Chrome DevTools Protocol to call the … | 59 | 1554 | active |
| zohaibbashir/Google-Maps-Scrapper A Python CLI script built on Playwright that scrapes Google Maps listings to extract business details such as name, address, website, phone… | 62 | 1552 | active |
| Rhizome-Conifer/conifer Conifer is an open-source web archiving platform for capturing, replaying, and sharing collections of archived web pages through a user-fri… | 70 | 1549 | active |
| ecmadao/hacknical Hacknical is a web application that analyzes a GitHub user's data (contributions, commits, languages, repos) and helps generate a better de… | 77 | 1543 | active |
| yujiosaka/headless-chrome-crawler A Node.js library providing a distributed web crawler powered by Headless Chrome via Puppeteer. It can crawl JavaScript-rendered (SPA) webs… | 23 | 5633 | maintenance |
| LifeActor/ykdl YouKuDownLoader (ykdl) is a Python command-line video downloader focused on China mainland video sites, forked from you-get with restructur… | 44 | 1539 | active |
| xnl-h4ck3r/GAP-Burp-Extension GAP is a Burp Suite extension written in Python (Jython) that extracts potential endpoints, parameters, and links from Burp's site map, pro… | 60 | 1533 | active |
| orangecoding/fredy Fredy is a self-hosted Node.js application that continuously scrapes European real estate portals like ImmoScout24, Immowelt, Kleinanzeigen… | 93 | 1528 | active |
| rbren/rss-parser rss-parser is a lightweight JavaScript library that converts RSS and Atom XML feeds into JavaScript objects. It works in both Node.js and t… | 73 | 1528 | active |
| justfoolingaround/animdl animdl is a lightweight Python CLI tool that scrapes, streams, and downloads anime episodes from supported providers. It supports quality s… | 32 | 1525 | active |
| jlesage/docker-jdownloader-2 A Docker container packaging JDownloader 2, a download manager, with a browser-accessible graphical interface via web or VNC. It is an unof… | 97 | 1523 | active |
| tidyverse/rvest rvest is an R package from the tidyverse for scraping (harvesting) data from web pages, inspired by Beautiful Soup and RoboBrowser. It prov… | 66 | 1522 | active |
| SpiderClub/haipproxy A high-availability distributed IP proxy pool built with Scrapy and Redis that scrapes free proxies from the internet, validates them, and … | 23 | 5525 | maintenance |
| SilentDemonSD/WZML-X WZML-X is a self-hosted Telegram mirror and leech bot written in Python that downloads files from torrents, Mega, Google Drive, direct link… | 93 | 1513 | active |
| misiektoja/instagram_monitor A Python-based OSINT tool that tracks Instagram users' activities in real time, including story updates, profile changes, and follower shif… | 90 | 1512 | active |
| exa-labs/company-researcher An open-source web application by Exa.ai that lets users enter a company URL and instantly gathers comprehensive research about it, includi… | 66 | 1511 | active |
| JustAnotherArchivist/snscrape snscrape is a Python-based scraper for social networking services that extracts posts, profiles, hashtags, and search results from platform… | 32 | 5464 | maintenance |
| manojVivek/medium-unlimited A browser extension for Chrome and Firefox that unlocks Medium.com membership-only articles by bypassing the paywall. It supports medium.co… | 10 | 5462 | maintenance |
| nickclyde/duckduckgo-mcp-server A Model Context Protocol (MCP) server that exposes DuckDuckGo web search to LLM clients like Claude Desktop and Claude Code. It also fetche… | 83 | 1501 | active |
| apinanaivot/IKEA-3D-Model-Download-Button A Tampermonkey userscript that adds a 'Download 3D' button to IKEA product pages, letting users save the product's 3D model as a .GLB file.… | 50 | 1501 | active |
| Vincentqyw/cv-arxiv-daily An automated daily digest of computer vision and robotics arXiv papers (SLAM, SFM, visual localization, keypoint detection, image matching,… | 76 | 1500 | active |
| AI4Finance-Foundation/FinNLP FinNLP is a Python library for collecting internet-scale financial data from sources like Finnhub, Yahoo Finance, Reuters, and Sina Finance… | 31 | 1491 | active |
| dteviot/WebToEpub A Chrome and Firefox browser extension that converts web novels and other web pages into EPUB ebooks for offline reading. It supports many … | 92 | 1490 | active |
| Tsuk1ko/bilibili-live-chat A backend-free web app that displays Bilibili live stream danmaku (chat messages) and gifts in a YouTube Live Chat style overlay, primarily… | 56 | 1490 | active |
| submato/xhscrawl A Python-based reverse-engineering toolkit for Xiaohongshu (XHS) web APIs, focusing on generating the encrypted x-s signature parameter via… | 69 | 1485 | active |
| karust/openserp OpenSERP is a self-hosted, MIT-licensed SERP API and CLI written in Go that returns structured search results from Google, Bing, Yandex, Ba… | 93 | 1484 | active |
| kepano/clipper-templates A collection of templates for the Obsidian Web Clipper browser extension, covering generic schemas (recipes, products) and specific sites l… | 52 | 1484 | active |
| asz798838958/aBaiFreeGPT A self-hosted account registration and lifecycle management platform that automates bulk account signup, email verification, TOTP 2FA bindi… | 55 | 1482 | active |
| SEO Analyzer A Python-based SEO analyzer that crawls a website, analyzes its structure, counts body words, and reports technical SEO issues, with option… | 65 | 1481 | active |
| eliasdabbas/advertools advertools is a Python library of productivity and analysis tools for online/digital marketing, built as a set of independent, composable f… | 69 | 1477 | active |
| Vega Vega is an open-source Android media streaming app built with React Native and TypeScript that lets users stream and download video content… | 90 | 1475 | active |
| Surfer-Org/Protocol Surfer Protocol is an open-source framework for exporting personal data from platforms like Gmail, iMessages, Twitter, Notion, and ChatGPT.… | 14 | 1474 | active |
| AgentQL AgentQL is a suite of tools for extracting structured data and automating workflows on live websites using an AI-powered natural language q… | 70 | 1472 | active |
| epsylon/xsser XSSer is an automatic penetration testing framework for detecting, exploiting, and reporting cross-site scripting (XSS) vulnerabilities in … | 80 | 1470 | active |
| momosecurity/FindSomething FindSomething is a passive browser extension for Chrome and Firefox that extracts potentially sensitive information (like emails, API keys,… | 32 | 1470 | active |
| shuanx/BurpAPIFinder BurpAPIFinder is a Burp Suite extension written in Java that passively analyzes HTTP traffic (HTML and JS files) to discover hidden API end… | 16 | 1469 | active |
| lanlinju/Animius Animius is a clean, minimalist Android app for watching anime, built with Jetpack Compose and Kotlin. It supports danmaku (bullet comments)… | 76 | 1464 | active |
| denandz/sourcemapper Sourcemapper is a Go CLI tool that parses JavaScript sourcemap (.map) files, whether from URLs or local directories, and reconstructs the o… | 71 | 1464 | stable |
| txperl/PixivBiu PixivBiu is a self-hosted Pixiv client written in Go that provides a browser-based interface for searching, filtering, browsing, and downlo… | 93 | 1462 | active |
| hxh19950701/WebViewTvLive An Android TV live streaming app that loads official broadcaster web pages in a Tencent X5 WebView, auto-fullscreens the video element, and… | 85 | 5288 | maintenance |
| mylar3/mylar3 Mylar3 is a Python-based automated comic book (cbr/cbz) downloader that monitors a watchlist of series and grabs new issues via NZB indexer… | 60 | 1460 | active |