Open-source web-scraping software
2000 open-source libraries, tools and applications tagged web-scraping, ranked by how closely web-scraping is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
2000 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| pystardust/ani-cli ani-cli is a POSIX shell script CLI tool for browsing and watching anime from the terminal, scraping the anidb site and playing videos via … | 96 | 13851 | active |
| mealie-recipes/mealie Mealie is a self-hosted recipe manager and meal planner with a FastAPI REST backend and a Vue frontend. It imports recipes from URLs via a … | 96 | 13611 | active |
| instaloader/instaloader Instaloader is a Python command-line tool and library for downloading pictures, videos, captions, comments, and metadata from Instagram. It… | 91 | 13598 | active |
| HackerNews/API Official documentation and samples for the Hacker News public API, hosted on Firebase. It describes how to access HN items (stories, commen… | 32 | 13357 | stable |
| chromedp chromedp is a Go library for driving Chrome and other browsers via the Chrome DevTools Protocol, with no external dependencies. It supports… | 98 | 13301 | active |
| seleniumbase/SeleniumBase SeleniumBase is an all-in-one Python browser automation framework for web testing, crawling, and scraping, built on Selenium/WebDriver with… | 93 | 13040 | active |
| JoeanAmier/XHS-Downloader A tool for extracting links and downloading content (images, videos, live photos) from XiaoHongShu (RedNote) posts. It ships as a Python TU… | 86 | 13039 | active |
| wallabag wallabag is a self-hostable web application for saving web pages to read later, extracting article content while stripping ads and distract… | 75 | 12994 | stable |
| ultrafunkamsterdam/undetected-chromedriver A Python library that patches Selenium's ChromeDriver binary so automated Chrome sessions avoid detection by anti-bot systems like Cloudfla… | 42 | 12858 | active |
| Pocket Flow An AI agent built with the Pocket Flow LLM framework that crawls GitHub repositories, analyzes codebases to identify core abstractions, and… | 55 | 12683 | active |
| daijro/camoufox Camoufox is an open-source anti-detect browser built on Firefox for web scraping and AI agents, with browser fingerprint spoofing and Playw… | 91 | 12641 | active |
| elebumm/RedditVideoMakerBot A Python application that automatically generates Reddit story videos for platforms like TikTok, YouTube, and Instagram with a single comma… | 70 | 12541 | active |
| g1879/DrissionPage DrissionPage is a Python-based web automation library that combines browser control (via Chromium's DevTools Protocol, without WebDriver) w… | 70 | 12511 | active |
| hangwin/mcp-chrome A Chrome extension that acts as a Model Context Protocol (MCP) server, exposing your real Chrome browser—its tabs, login states, and config… | 50 | 12496 | active |
| jiji262/douyin-downloader A Python-based Douyin (TikTok China) downloader that fetches videos, image galleries, collections, and music without watermarks, supporting… | 96 | 12332 | active |
| Crawlab Crawlab is a Go-based distributed web crawler management platform with a web UI for managing, scheduling, and monitoring spiders written in… | 49 | 12278 | active |
| jina-ai/reader Jina AI Reader converts any URL into LLM-friendly markdown via the r.jina.ai prefix, and searches the web into markdown via s.jina.ai. It r… | 59 | 12158 | active |
| seanmonstar/reqwest Reqwest is an ergonomic, batteries-included HTTP client library for Rust built on hyper. It supports async and blocking clients, JSON/multi… | 96 | 11845 | stable |
| Shiori Shiori is a simple bookmark manager written in Go, intended as a self-hosted clone of Pocket. It works as both a command-line application a… | 79 | 11670 | active |
| code4craft/webmagic WebMagic is a scalable web crawler framework for Java covering the full crawl lifecycle: downloading, URL management, content extraction (X… | 56 | 11670 | active |
| jo-inc/camofox-browser A self-hosted anti-detection browser server for AI agents, wrapping the Camoufox Firefox fork that spoofs fingerprints at the C++ level. It… | 83 | 11663 | active |
| mozilla/readability Mozilla's standalone Readability library that extracts the main article content, title, and metadata from HTML documents, powering Firefox … | 71 | 11453 | stable |
| jhy/jsoup jsoup is a Java library for parsing, manipulating, and cleaning real-world HTML and XML, implementing the WHATWG HTML5 specification to pro… | 93 | 11410 | stable |
| ihmily/DouyinLiveRecorder A Python-based livestream recording application that can run unattended in a loop and record multiple streams simultaneously from 40+ platf… | 59 | 11051 | active |
| zubair-trabzada/geo-seo-claude A Claude Code skill that audits and optimizes websites for AI-powered search engines (ChatGPT, Perplexity, Gemini, Google AI Overviews) alo… | 60 | 10928 | active |
| Thysrael/Horizon Horizon is an AI-powered news aggregation platform that fetches content from sources like Hacker News, GitHub, RSS, Reddit, and Telegram, s… | 61 | 10909 | active |
| BBOT BBOT is a recursive, multipurpose internet scanner built in Python for automating OSINT reconnaissance, bug bounty hunting, and attack surf… | 97 | 10709 | active |
| Ranchero-Software/NetNewsWire NetNewsWire is a free and open-source RSS/Atom/JSON Feed reader for macOS and iOS. It displays articles from blogs and news sites, tracks r… | 97 | 10413 | active |
| pinchtab/pinchtab PinchTab is a standalone Go HTTP server that gives AI agents direct control over Chrome via CDP, with stealth injection and multi-instance … | 79 | 10299 | active |
| shmilylty/OneForAll OneForAll is a powerful Python-based subdomain collection and enumeration tool for reconnaissance. It gathers subdomains via brute forcing,… | 56 | 10094 | active |
| thewhiteh4t/seeker Seeker is a security testing tool that hosts a fake website asking users for browser location permission, capturing GPS coordinates (longit… | 68 | 10055 | active |
| ltaoo/wx_channels_download A small desktop application that downloads videos from WeChat Channels (视频号) by injecting a download button into the WeChat PC client via a… | 88 | 9927 | active |
| Apify Agent Skills The Apify MCP Server exposes thousands of Apify Store scrapers, crawlers, and automation tools to AI agents via the Model Context Protocol,… | 83 | 9896 | active |
| FongMi/TV An open-source Android video streaming app built on CatVod that supports both Android TV (leanback) and mobile UIs, with VOD browsing, live… | 77 | 9844 | active |
| dataabc/weiboSpider A Python command-line crawler that scrapes posts and profile data from Sina Weibo users, writing results to txt/csv/json files or MySQL/Mon… | 58 | 9733 | active |
| city-roads A web application that renders every road in any city at once as an interactive visualization, using data from OpenStreetMap. It includes a… | 61 | 9725 | active |
| agefanscom/website A release page that publishes the current official website URLs and app download links for AGE Animation (AGE动漫), an anime streaming site w… | 70 | 9724 | active |
| kepano/defuddle Defuddle is a TypeScript library, CLI, and hosted service that extracts the main content from web pages, removing clutter like comments, si… | 88 | 9723 | active |
| cooderl/wewe-rss WeWe RSS is a self-hostable service that generates RSS feeds for WeChat official accounts by leveraging WeRead as a data source. It support… | 10 | 9650 | active |
| qiye45/wechatDownload A desktop tool for batch downloading WeChat Official Account (公众号) articles, saving them as html/mhtml/md/pdf/docx/csv and preserving embed… | 84 | 9560 | active |
| ntegrals/openbrowser Open Browser is a TypeScript framework that lets AI agents autonomously control a web browser to complete tasks like clicking, typing, navi… | 62 | 9542 | active |
| ccfddl/ccf-deadlines A community-maintained tracker of worldwide academic conference deadlines, offering a website portal, WeChat applet, and multiple extension… | 77 | 9463 | active |
| XiaoYouChR/Ghost-Downloader-3 Ghost Downloader is a cross-platform, open-source download manager built with Python and PySide6/Qt that handles HTTP, BitTorrent/magnet, F… | 92 | 9416 | active |
| jiangrui1994/CloudSaver CloudSaver is a self-hosted web application for searching cloud drive (netdisk) resources across multiple sources and transferring them to … | 67 | 9380 | active |
| Legado (阅读) Legado (阅读 3.0) is an open-source Android e-book reader that lets users define custom book sources to read web novels and online content. I… | 57 | 47112 | maintenance |
| RSS-Bridge/rss-bridge RSS-Bridge is a PHP web application that generates RSS, Atom, and JSON feeds for websites that don't offer one, using hundreds of site-spec… | 75 | 9295 | active |
| mediago-dev/mediago MediaGo is a cross-platform video downloader that automatically sniffs m3u8/HLS streams and other video resources from web pages, supportin… | 77 | 9237 | active |
| everywall/ladder Ladder is a self-hosted Go web proxy that removes CORS and other headers (like CSP) and modifies HTML/CSS/JS in responses, serving as an al… | 84 | 9173 | active |
| eze-is/web-access An Agent Skill that gives AI coding agents (Claude Code, Cursor, Gemini CLI, etc.) full web access capabilities: three-tier channel dispatc… | 63 | 9124 | active |
| loks666/get_jobs An AI-powered job application assistant that automatically submits resumes across Chinese job platforms (Boss Zhipin, 51job, Liepin, Zhaopi… | 81 | 9081 | active |
| BiliBiliTool BiliTool is a .NET-based automated task tool for Bilibili that performs scheduled tasks like daily check-ins, coin donations, lottery parti… | 100 | 9027 | active |
| HerbertHe/iptv-sources A service that automatically aggregates and updates IPTV m3u playlist sources from multiple public repositories, with EPG data and optional… | 72 | 9026 | active |
| p1ngul1n0/blackbird Blackbird is a Python CLI OSINT tool that searches for user accounts by username or email across 600+ social networks and platforms, levera… | 43 | 8982 | active |
| reconurge/flowsint Flowsint is an open-source, self-hosted OSINT platform for visual, graph-based investigations, providing entity relationship visualization … | 84 | 8937 | active |
| Builder.io GPT Crawler is a TypeScript‑based CLI that crawls websites, extracts content, and formats it into knowledge files for OpenAI custom GPTs. I… | 94 | 8862 | active |
| Sitoi/dailycheckin DailyCheckIn is a Python-based daily check-in script collection for Chinese websites and services (Bilibili, Baidu Tieba, iQiyi, V2EX, AcFu… | 68 | 8720 | active |
| hardikvasa/google-images-download A Python command-line tool that searches and downloads hundreds of images from Google Images to local storage. It uses Selenium with Chrome… | 66 | 8695 | active |
| Tube Archivist Tube Archivist is a self-hosted YouTube media server that downloads videos via yt-dlp, indexes them with metadata in Elasticsearch, and ser… | 89 | 8483 | active |
| freeok/so-novel So Novel is a Java-based tool for extracting structured content from web pages and exporting it as EPUB, TXT, or PDF ebooks. It offers CLI,… | 88 | 8418 | active |
| mherrmann/helium Helium is a Python library that provides a high-level, human-friendly API for browser automation on top of Selenium, supporting Chrome and … | 85 | 8318 | active |
| EstrellaXD/Auto_Bangumi AutoBangumi is a self-hosted, RSS-based automated anime downloading and organizing tool. It parses RSS feeds from sites like Mikan Project,… | 99 | 8285 | active |
| kangvcar/InfoSpider InfoSpider is an open-source Python toolbox that crawls a user's own personal data from dozens of Chinese and international services (email… | 54 | 8264 | active |
| epi052/feroxbuster feroxbuster is a fast, recursive content discovery tool written in Rust that performs forced browsing against web servers. It brute-forces … | 77 | 8112 | active |
| cv-cat/Spider_XHS A Python library that reverse-engineers Xiaohongshu (Little Red Book) signature algorithms and wraps the platform's PC, creator, and Pugong… | 89 | 8053 | active |
| alirezamika/autoscraper AutoScraper is a Python library that automatically learns scraping rules from a URL or HTML content plus a list of sample data you want to … | 62 | 8003 | stable |
| adithya-s-k/omniparse OmniParse is a self-hosted ingestion and parsing platform that converts unstructured data (documents, images, audio, video, web pages) into… | 46 | 7932 | active |
| Suwayomi (Tachidesk) Suwayomi (formerly Tachidesk) is a free, self-hosted manga reader server that is a rewrite of Tachiyomi for desktop platforms. It runs a se… | 99 | 7905 | active |
| Bili23 Downloader Bili23-Downloader is a free, open-source, cross-platform desktop application for downloading videos from Bilibili. It supports multithreade… | 98 | 7796 | active |
| Steel Browser Steel Browser is an open-source browser API and sandbox that manages browser sessions, proxies, stealth, and lifecycle so developers can bu… | 86 | 7708 | active |
| NomaDamas/k-skill A collection of ~80 LLM agent skills tailored for Korean users, installable into coding agents like Claude Code, Codex, and OpenCode. Skill… | 62 | 7672 | active |
| samuelclay/NewsBlur NewsBlur is an open-source personal RSS feed reader and social news network with intelligence training, full-text search, story archiving, … | 96 | 7661 | active |
| hect0x7/JMComic-Crawler-Python A Python library providing an API client for the JMComic (18comic) site, supporting both web and mobile endpoints, with album downloading, … | 98 | 7607 | stable |
| eRPC Pholcus is a distributed, high-concurrency web crawler framework written in pure Go. It supports standalone, server, and client modes with … | 84 | 7582 | active |
| mgdm/htmlq htmlq is a command-line tool, like jq but for HTML, that extracts content from HTML documents using CSS selectors. Written in Rust, it read… | 57 | 7580 | stable |
| metafizzy/infinite-scroll Infinite Scroll is a JavaScript plugin that automatically loads and appends the next page of content as the user scrolls, avoiding full pag… | 24 | 7477 | stable |
| TheCraigHewitt/seomachine SEO Machine is a specialized Claude Code workspace of custom commands, AI agents, and Python analysis modules for researching, writing, and… | 57 | 7467 | active |
| libcpr/cpr cpr (C++ Requests) is a modern C++ HTTP client library that wraps libcurl with a simple, Python Requests-inspired API. It supports GET/POST… | 95 | 7457 | active |
| apurvsinghgautam/robin Robin is an AI-powered dark web OSINT investigation tool that uses LLMs to refine queries, filter results from dark web search engines, and… | 86 | 7440 | active |
| mvdctop/Movie_Data_Capture A Python-based local movie organizer that scrapes and renames movie metadata for media servers. It prepares local movie files with correct … | 95 | 7431 | active |
| symfony/css-selector A Symfony PHP component that converts CSS selectors into XPath expressions. It lets developers query HTML or XML documents using familiar C… | 94 | 7419 | stable |
| berstend/puppeteer-extra A modular plugin framework for Puppeteer (and Playwright via playwright-extra) that extends headless browser automation with drop-in plugin… | 32 | 7406 | active |
| RSSHub RSSHub Radar is a browser extension that helps users quickly discover RSS and RSSHub feeds available on the web pages they visit. It simpli… | 72 | 7366 | active |
| WECENG/ticket-purchase A Python automation tool for snatching tickets on Damai (大麦), China's major event ticketing platform, using Selenium for the web and Appium… | 61 | 7306 | active |
| BrowserMCP/mcp Browser MCP is an MCP server plus Chrome extension that lets AI applications like Claude, Cursor, VS Code, and Windsurf control the user's … | 25 | 7171 | active |
| happycola233/tchMaterial-parser A cross-platform GUI tool that parses and downloads electronic textbook PDFs from China's National Smart Education Platform for Primary and… | 95 | 7158 | active |
| Rod Rod is a high-level Go library that drives Chrome via the Chrome DevTools Protocol for web automation and scraping. It offers both high-lev… | 63 | 7132 | active |
| autoscrape-labs/pydoll Pydoll is a Python library for automating Chromium-based browsers directly over the Chrome DevTools Protocol, with no WebDriver binary and … | 84 | 7107 | active |
| Momo707577045/m3u8-downloader A browser-based tool for extracting and downloading m3u8 (HLS) streaming videos by fetching the m3u8 playlist, downloading .ts segments in … | 36 | 7080 | active |
| InfinityLoop1308/PipePipe PipePipe is an open-source Android app, forked from NewPipe, for browsing and streaming YouTube, BiliBili, and NicoNico without ads, tracke… | 100 | 7068 | active |
| subzeroid/instagrapi instagrapi is a fast Python wrapper for Instagram's unofficial private (mobile) and public web APIs, supporting login with 2FA, session per… | 96 | 6983 | active |
| InternLM/MindSearch MindSearch is an open-source, LLM-based multi-agent framework that mimics human cognition to perform deep web searches, similar to Perplexi… | 24 | 6943 | active |
| mishushakov/llm-scraper A TypeScript library that turns any webpage into structured data using LLMs, built on Playwright and the Vercel AI SDK. It supports multipl… | 68 | 6937 | active |
| adbar/trafilatura Trafilatura is a Python package and command-line tool for crawling the web and extracting main text, metadata, and comments from raw HTML w… | 94 | 6914 | active |
| webclipper/web-clipper Web Clipper is an open-source browser extension that saves web content to many note-taking platforms such as Notion, OneNote, Obsidian, Bea… | 43 | 6856 | active |
| bda-research/node-crawler node-crawler is a TypeScript web crawler/spider library for Node.js that fetches pages and provides server-side DOM parsing with automatic … | 84 | 6794 | active |
| VeNoMouS/cloudscraper A Python library that wraps Requests to bypass Cloudflare's anti-bot protection pages (IUAM), supporting challenge types v1, v2, v3, and Tu… | 30 | 6776 | active |
| wzdnzd/aggregator A Python-based platform that crawls free proxy nodes from sources like Telegram, GitHub, and search engines, validates their liveness, and … | 75 | 6774 | active |
| curl-impersonate bindings curl_cffi is a Python binding for a curl-impersonate fork via cffi, providing an HTTP client that can impersonate browser TLS/JA3, HTTP/2, … | 99 | 6746 | active |
| davidteather/TikTok-Api An unofficial Python API wrapper for TikTok.com that retrieves trending content, user information, hashtags, and video data without authent… | 85 | 6681 | active |
| SawyerHood/dev-browser A CLI tool that lets AI agents and developers control a browser by running sandboxed JavaScript scripts against a full Playwright API. It s… | 77 | 6636 | active |