Open-source web-scraping software
2000 open-source libraries, tools and applications tagged web-scraping, ranked by how closely web-scraping is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
2000 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| watsonbox/exportify Exportify is a browser-based application for exporting and backing up Spotify playlists to CSV files using the Spotify Web API. It runs ent… | 61 | 4238 | active |
| Seanime Seanime is an open-source self-hosted media server for anime and manga, offering a web interface and desktop app to manage local libraries,… | 88 | 4221 | active |
| InstaPy InstaPy is a Python library built on Selenium that automates Instagram interactions such as liking, commenting, following, and unfollowing … | 67 | 18309 | maintenance |
| zly2006/zhihu-plus-plus Zhihu++ is an open-source third-party Android client for the Chinese Q&A platform Zhihu, built in Kotlin, that removes ads, promotional pos… | 84 | 4184 | active |
| ivre/ivre IVRE is an open-source network recon framework written in Python that collects, stores, and analyzes network intelligence from active scann… | 66 | 4177 | active |
| 027xiguapi/code-box CodeBox is a browser extension for Chrome, Edge, Firefox, and 360 browsers that enhances Chinese tech blog sites like CSDN, Zhihu, Juejin, … | 57 | 4150 | active |
| dotnetcore/DotnetSpider DotnetSpider is a .NET Standard web crawling and scraping framework that is lightweight, efficient, and cross-platform. It supports distrib… | 53 | 4144 | active |
| RipMeApp/ripme RipMe is a cross-platform Java application that bulk-downloads image albums from websites like Reddit, Imgur, Twitter, Instagram, and Tumbl… | 92 | 4126 | active |
| ericciarla/trendFinder A self-hosted Node.js application that monitors influencer posts on Twitter/X and website changes via Firecrawl, then uses LLMs (Together A… | 22 | 4120 | active |
| nghuyong/WeiboSpider A continuously maintained Python web scraping tool for Sina Weibo built on Scrapy and the new weibo.com API. It collects user profiles, pos… | 69 | 4119 | active |
| tonquer/JMComic-qt A cross-platform desktop client for the JMComic (18comic) comic site, built with Python and Qt (PySide6). It supports browsing, searching, … | 93 | 4107 | active |
| DailyHot API DailyHotApi is a TypeScript-based API service that aggregates trending/hot list data from many Chinese platforms (Bilibili, Weibo, Zhihu, D… | 53 | 4088 | active |
| libredirect/browser_extension LibRedirect is a browser extension (WebExtension) that automatically redirects requests to popular sites like YouTube, Twitter, Reddit, Ins… | 87 | 4081 | active |
| React Native Upgrade Helper A web application that shows the exact file diffs between any two React Native versions to guide app upgrades. It is built on the rn-diff-p… | 70 | 4072 | active |
| bdvajstudio/javdb The official mobile app for JavDB, an adult video database website cataloging Japanese and Western adult films with search, tagging, rating… | 61 | 4072 | active |
| IonicaBizau/scrape-it scrape-it is a Node.js web scraping library with a simple, declarative API for extracting data from HTML pages, built on top of tinyreq and… | 99 | 4069 | active |
| ipcjs/oh-my-userscripts A collection of Tampermonkey/Greasyfork userscripts by ipcjs that tweak and enhance websites like Bilibili, Zhihu, Bangumi, S1, and Google.… | 77 | 4065 | active |
| yacy/yacy_search_server YaCy is a full search engine application in Java that combines a web crawler, a search index server, and a web front-end. It can run standa… | 75 | 4053 | active |
| fake-useragent/fake-useragent A Python library that generates realistic, up-to-date browser user-agent strings from a bundled real-world database. It supports random or … | 10 | 4049 | active |
| nkanaev/yarr yarr is a web-based RSS feed aggregator written in Go, distributed as a single dependency-free binary that runs as a desktop app or self-ho… | 88 | 4046 | active |
| snooppr/snoop Snoop is a Python-based OSINT CLI tool that searches for a given username/nickname across ~5400+ websites, with a focus on the CIS region. … | 70 | 4036 | active |
| symfony/dom-crawler Symfony DomCrawler is a PHP component that eases DOM navigation for HTML and XML documents. It provides a Crawler class for querying and tr… | 93 | 4027 | stable |
| mahdibland/V2RayAggregator A Python-based automation that aggregates free proxy nodes (Shadowsocks, SSR, Trojan, Vmess) from public sources, deduplicates them, and sp… | 68 | 4024 | active |
| MikeChongCan/scylla Scylla is an intelligent proxy pool service that automatically crawls, validates, and serves proxy IPs via a JSON API and web UI. It integr… | 30 | 4019 | active |
| DaKheera47/job-ops A self-hosted job search assistant that aggregates searches across 10+ job boards, AI-scores role fit, tailors CVs per posting, and tracks … | 82 | 4004 | active |
| zenbu-labs/terminal-browser terminal-browser is a real Chromium-based browser that renders inside your terminal using the kitty graphics protocol and Electron's offscr… | 81 | 3933 | active |
| JoeanAmier/TikTokDownloader DouK-Downloader (formerly TikTokDownloader) is an open-source Python tool for downloading and scraping content from Douyin and TikTok, incl… | 88 | 16930 | maintenance |
| hardkoded/puppeteer-sharp PuppeteerSharp is a .NET port of the official Node.js Puppeteer API for controlling headless or headful Chrome and Firefox. It supports nav… | 100 | 3928 | active |
| DO-SAY-GO/tdf TDF (Tree Document Format) is a pair of native desktop apps for capturing live web browsing sessions into portable, offline-replayable .tdf… | 67 | 3910 | active |
| binux/pyspider pyspider is a powerful web crawler (spider) system written in Python with a built-in WebUI for script editing, task monitoring, project man… | 10 | 16755 | maintenance |
| zhongbai2333/Tomato-Novel-Downloader A Rust-based downloader for Tomato (Fanqie) novels that exports books as EPUB and other formats, with TUI, Web UI, and legacy CLI interface… | 78 | 3894 | active |
| uncle-novel/uncle-novel Uncle Novel is a cross-platform desktop novel/e-book reader application built with Java and OpenJFX, supporting macOS and Windows. It lets … | 52 | 3879 | active |
| pyload/pyload pyLoad is a free, open-source download manager written in pure Python, designed to be lightweight and extensible. It automates downloads fr… | 72 | 3877 | active |
| Tyrrrz/YoutubeDownloader YoutubeDownloader is a cross-platform desktop GUI application for downloading videos, playlists, and channels from YouTube in formats like … | 97 | 16336 | maintenance |
| kingname/GeneralNewsExtractor GNE (GeneralNewsExtractor) is a Python library that extracts article title, author, publish time, body text, and images from news page HTML… | 75 | 3802 | active |
| stickerdaniel/linkedin-mcp-server An open-source MCP server that gives AI assistants like Claude and ChatGPT access to LinkedIn data (profiles, companies, jobs, messages) th… | 83 | 3800 | active |
| Omnivore Omnivore is a complete, open source read-it-later application for saving and reading articles, with highlighting, notes, search, labels, PD… | 95 | 16257 | maintenance |
| Homelable Homelable is a self-hosted web application for visualizing homelab infrastructure on interactive network, electrical, and 19-inch rack canv… | 80 | 3796 | active |
| JavScraper/Emby.Plugins.JavScraper A metadata scraper plugin for Emby and Jellyfin media servers that fetches Japanese adult movie information (titles, covers, tags, actress … | 23 | 3794 | active |
| edoardottt/cariddi Cariddi is a fast command-line web crawler written in Go that takes a list of domains, crawls URLs, and scans for endpoints, secrets, API k… | 84 | 3786 | active |
| flutter_inappwebview A Flutter plugin for embedding inline WebView widgets, running headless WebViews, and opening in-app browser windows (Chrome Custom Tabs / … | 57 | 3779 | active |
| ibnaleem/gosearch GoSearch is a Go-based CLI OSINT tool that searches a username across 300+ websites to find a person's digital footprint, serving as a Sher… | 82 | 3775 | active |
| foamzou/melody Melody is a self-hosted web application ('your music genie') that helps users search for songs across music and video platforms (NetEase, Q… | 61 | 3737 | active |
| feapder feapder is a powerful, easy-to-use Python web crawling framework offering four spider types (AirSpider, Spider, TaskSpider, BatchSpider) fo… | 68 | 3736 | active |
| niumoo/bing-wallpaper A Java-based project that automatically fetches Bing's daily 4K wallpaper and archives it with a browsable web gallery and API. It publishe… | 75 | 3730 | active |
| KitchenOwl KitchenOwl is a self-hosted grocery list and recipe manager with native mobile, web, and desktop apps built in Flutter and a Flask backend.… | 94 | 3717 | active |
| EhViewer-NekoInverter/EhViewer An Android client app for browsing E-Hentai/ExHentai galleries, forked from EhViewer with a classic Material Design 2 style. It is maintain… | 78 | 3708 | active |
| leaperone/MultiPost-Extension MultiPost is an open-source browser extension (with a companion desktop app) that publishes text, images, videos, and other content to 10+ … | 88 | 3670 | active |
| waditu/tushare TuShare is a Python library for crawling, cleaning, and storing historical and realtime financial data for China stocks and futures. It pro… | 23 | 15438 | maintenance |
| oxylabs/google-ai-mode-scraper A code repository of examples for Oxylabs' Google AI Mode Scraper, a commercial API that sends prompts to Google AI Mode and returns parsed… | 59 | 3626 | active |
| zc-zhangchen/any-auto-register A multi-platform automated account registration and management system with a Web UI, plugin-based extensibility, and batch registration sup… | 58 | 3625 | active |
| Dembrandt Dembrandt is an open-source Node.js CLI that extracts a website's design system—colors, typography, spacing, borders, shadows, motion, comp… | 80 | 3624 | active |
| webtorrent/instant.io Instant.io is a web application for streaming file transfer over WebTorrent (BitTorrent over WebRTC), letting users share and download file… | 76 | 3595 | active |
| s0md3v/XSStrike XSStrike is a Python command-line Cross Site Scripting (XSS) detection suite that uses hand-written HTML/JavaScript parsers, context analys… | 27 | 15197 | maintenance |
| codelucas/newspaper newspaper3k is a Python 3 library for discovering, downloading, and parsing news articles from websites. It extracts full text, authors, pu… | 66 | 15174 | maintenance |
| jstrieb/github-stats A tool that generates GitHub profile and repository statistics visualizations as SVG images using GitHub Actions, including data from priva… | 91 | 3561 | active |
| elliotgao2/toapi A Python library that turns any website into a JSON API by declaring fields with CSS/XPath selectors. It fetches and parses pages on demand… | 73 | 3556 | active |
| wushuo894/ani-rss ANI-RSS is a self-hosted Java application that automates anime tracking via RSS feeds: it subscribes to shows, downloads torrents through q… | 87 | 3555 | active |
| peasoft/NoMoreWalls A Python project that automatically fetches and merges publicly available proxy nodes from the internet and republishes them as Base64 and … | 77 | 3555 | active |
| ytmdl ytmdl is a Python CLI tool (with an optional web app) that downloads songs from YouTube as MP3s and enriches them with metadata like artist… | 23 | 3546 | active |
| smallfawn/QLScriptPublic A public collection of JavaScript automation scripts for the Qinglong (青龙) task-scheduling panel, primarily performing daily check-ins, sig… | 75 | 3543 | active |
| omkarcloud/google-maps-scraper A desktop application and API for scraping Google Maps business data, extracting 50+ data points including emails, phone numbers, social pr… | 73 | 3538 | active |
| pjialin/py12306 A Python-based ticket booking assistant for China's 12306 railway system, supporting multi-account, multi-task ticket purchasing with distr… | 66 | 14959 | maintenance |
| DeepSourceCorp/good-first-issue Good First Issue is a web application that curates beginner-friendly issues from popular open-source projects so new developers can make th… | 60 | 3534 | active |
| Gerapy Gerapy is a distributed crawler management framework built on Scrapy, Scrapyd, Django, and Vue.js. It provides a web dashboard for managing… | 60 | 3515 | active |
| thomasdondorf/puppeteer-cluster A Node.js library that manages a pool of Puppeteer-controlled Chromium instances for running browser tasks in parallel. It handles job queu… | 54 | 3515 | stable |
| mxschmitt/playwright-go Playwright for Go is a Go library that automates Chromium, Firefox, and WebKit browsers through a single API, supporting both headless and … | 90 | 3510 | active |
| stevenschobert/instafeed.js Instafeed.js is a lightweight JavaScript plugin that displays Instagram photos on a website. It fetches images via the Instagram API and re… | 36 | 3508 | active |
| oxylabs/free-proxy-list A repository promoting Oxylabs' free tier of US datacenter proxies (HTTP/HTTPS/SOCKS5), offering 5 US IPs, 20 concurrent sessions, and 5GB … | 51 | 3487 | active |
| growchief/growchief GrowChief is an open-source, self-hostable social media automation and outreach tool that automates actions like sending connection request… | 37 | 3486 | active |
| OCS 网课助手 OCS (Online Course Script) is a userscript and companion Electron desktop app that automates online course tasks for Chinese university e-l… | 88 | 3484 | active |
| MZCretin/RollToolsApi RollToolsApi is a free, long-maintained aggregated REST API service that provides commonly used data endpoints for developers, hosted on an… | 70 | 3479 | active |
| imraywang/wewrite WeWrite is a Python-based AI agent skill that automates the full WeChat Official Account content pipeline: topic selection from trending ne… | 77 | 3473 | active |
| white0dew/XiaohongshuSkills A Python CLI tool and agent Skill that automates Xiaohongshu (RED/RedNote) via Chrome DevTools Protocol, supporting publishing posts with i… | 60 | 3471 | active |
| Samueli924/chaoxing A Python command-line tool that automatically completes task points (videos, quizzes) on the Chaoxing Xuexitong / Fanya online learning pla… | 74 | 3469 | active |
| tamnd/kage kage is a Go CLI tool that clones websites into browsable offline folders by rendering each page in headless Chrome, snapshotting the final… | 73 | 3453 | active |
| gautamkrishnar/blog-post-workflow A GitHub Action that automatically fetches your latest blog posts, StackOverflow activity, or YouTube videos via RSS feeds and displays the… | 95 | 3446 | active |
| opsdisk/pagodo pagodo is a Python CLI tool that automates passive Google dork searches by scraping the Google Hacking Database (GHDB) and running those qu… | 68 | 3412 | active |
| Oxylabs Amazon Scraper A Python-based free tool and sample code for Oxylabs' Amazon Scraper API that extracts product, search, offer listing, reviews, Q&A, best s… | 68 | 3395 | active |
| oxylabs/oxylabs-ai-studio-py A Python SDK for Oxylabs AI Studio, a commercial API offering AI-powered web scraping, crawling, search, site mapping, and browser automati… | 10 | 3395 | active |
| IPTV A Node.js tool that downloads Electronic Program Guide (EPG) data for thousands of TV channels from hundreds of sources and outputs it as X… | 77 | 3388 | active |
| oxylabs/google-news-scraper A free Python-based command-line tool from Oxylabs that scrapes Google News articles by topic, exporting headlines, URLs, and publication d… | 61 | 3381 | active |
| H4ckForJob/dirmap Dirmap is an advanced web directory and file scanning tool written in Python, designed to be more powerful than DirBuster, Dirsearch, cansi… | 66 | 3375 | stable |
| oxylabs/chatgpt-scraper Code examples and documentation for Oxylabs' ChatGPT Scraper, a commercial Web Scraper API endpoint that sends prompts to ChatGPT and retur… | 61 | 3370 | active |
| lxf746/any-auto-register A desktop application (Mac/Windows, built on Python/FastAPI with an embedded React UI) that automates registration and lifecycle management… | 70 | 3364 | active |
| Kovah/LinkAce LinkAce is a self-hosted web application for collecting, organizing, and archiving bookmarks with tags, lists, and advanced search. It auto… | 99 | 3345 | active |
| postaddictme/instagram-php-scraper A PHP library that scrapes Instagram via its web version to fetch account information, photos, videos, stories, and comments, with optional… | 30 | 3341 | active |
| internetarchive/heritrix3 Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler written in Java. It crawls websites and… | 95 | 3334 | active |
| cv-cat/DouYin_Spider A Python-based Douyin (Chinese TikTok) reverse-engineering toolkit that exposes the platform's full API surface for data collection, live-s… | 82 | 3331 | active |
| liuzi6612/nav A lightweight, fully static personal navigation/start-page website built with Angular and ng-zorro-antd, with 800+ curated sites built in a… | 71 | 3330 | active |
| psf/requests-html A Python library that combines HTTP requests with intuitive HTML parsing, offering CSS selector and XPath support plus JavaScript rendering… | 23 | 13807 | maintenance |
| php-curl-class/php-curl-class PHP Curl Class is a PHP library that wraps PHP's cURL extension in a simple object-oriented API for sending HTTP requests. It supports GET,… | 100 | 3298 | active |
| apache/nutch Apache Nutch is a highly extensible and scalable open-source web crawler built on Apache Hadoop data structures. It supports batch crawling… | 77 | 3297 | stable |
| itsOwen/CyberScraper-2077 CyberScraper 2077 is an AI-powered web scraping application with a Streamlit GUI that uses OpenAI, Gemini, or local Ollama models to intell… | 68 | 3291 | active |
| aapatre/Automatic-Udemy-Course-Enroller-GET-PAID-UDEMY-COURSES-for-FREE A Python script that scrapes coupon sites for free Udemy course coupons and automatically enrolls the user in paid courses for free using S… | 55 | 3289 | active |
| pydata/pandas-datareader A Python library that extracts data from remote internet sources such as FRED, Fama/French, World Bank, OECD, and Eurostat directly into pa… | 87 | 3272 | active |
| assetnote/kiterunner Kiterunner is a fast content discovery tool written in Go that bruteforces files, folders, and API routes on web servers. It uses a dataset… | 60 | 3272 | stable |
| oxylabs/ai-crawler-py A Python client library for Oxylabs AI Studio's AI-Crawler, a commercial service that crawls websites from a starting URL, uses natural lan… | 10 | 3264 | active |
| mtvpls/MoonTVPlus MoonTVPlus is a self-hosted, enhanced fork of MoonTV that aggregates video from multiple sources into a web-based streaming player built wi… | 59 | 3251 | active |
| Barabama/FreeNodes A Python crawler that aggregates free proxy nodes (v2ray, Clash, vmess, vless, trojan, ss) from public websites and publishes them as auto-… | 73 | 3238 | active |