Open-source web-scraping software
2000 open-source libraries, tools and applications tagged web-scraping, ranked by how closely web-scraping is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
2000 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| yt-dlp yt-dlp is a feature‑rich command‑line tool for downloading audio and video from thousands of websites. It is a actively maintained fork of … | 96 | 195314 | active |
| Firecrawl Firecrawl is an open‑source web data API that can search, scrape, and interact with web pages at scale. It returns clean markdown, JSON, sc… | 86 | 191248 | active |
| HelloGitHub HelloGitHub is a web service that curates and shares interesting, entry‑level open‑source projects from GitHub. It publishes monthly newsle… | 98 | 180299 | active |
| Browser Use Browser Use is a Python library that lets large language models control a web browser just like a human, performing clicks, typing, and for… | 84 | 117325 | active |
| Playwright Playwright is a framework for web testing and browser automation that drives Chromium, Firefox, and WebKit through a single API, available … | 96 | 96681 | stable |
| puppeteer/puppeteer Puppeteer is a JavaScript/TypeScript library that provides a high‑level API to control Chrome or Firefox via the DevTools Protocol or WebDr… | 95 | 95714 | active |
| D4Vinci/Scrapling Scrapling is a high‑performance Python library for web scraping and crawling that adapts to site changes automatically. It includes stealth… | 87 | 86704 | active |
| Panniantong/Agent-Reach Agent Reach is a Python‑based CLI that equips AI agents with the ability to browse and search the internet across multiple platforms. It pr… | 74 | 85113 | active |
| unclecode/crawl4ai Crawl4AI is an open‑source Python library that provides an asynchronous, headless‑browser crawler and scraper optimized for LLM‑ready markd… | 89 | 84624 | active |
| NanmiCoder/MediaCrawler MediaCrawler is a Python-based multi-platform social media crawler that scrapes notes, videos, posts, and comments from Xiaohongshu, Douyin… | 72 | 65481 | active |
| scrapy/scrapy Scrapy is a fast, high-level web crawling and scraping framework for Python. It provides tools for defining spiders, extracting data, and e… | 97 | 64503 | active |
| FoundationAgents/OpenManus OpenManus is an open-source framework for building general-purpose AI agents, created by the MetaGPT team as an open alternative to Manus. … | 66 | 58639 | active |
| soimort/you-get You-Get is a tiny command-line utility written in Python to download media content (videos, audios, images) from the web when no other hand… | 64 | 56873 | active |
| Huginn Huginn is a self‑hosted automation platform that lets you build agents to monitor, scrape, and act on web events. It functions like a hacka… | 100 | 50066 | active |
| Huginn Huginn is a self-hosted system for building agents that perform automated tasks online, such as monitoring websites, watching Twitter, and … | 100 | 50066 | active |
| RSSHub RSSHub is an open-source RSS feed generator that turns websites, social media platforms, and apps (Twitter, YouTube, Telegram, Bilibili, et… | 99 | 46497 | active |
| NaiboWang/EasySpider EasySpider is an open‑source visual, no‑code web crawler and browser automation tool. It lets users design and run scraping tasks through a… | 83 | 44578 | active |
| vercel-labs/agent-browser agent-browser is a fast, native Rust command‑line interface for controlling Chrome via the Chrome DevTools Protocol. It is designed for AI … | 79 | 43781 | active |
| BettaFish BettaFish (微舆) is a from-scratch multi-agent public opinion analysis system in Python that monitors 30+ social media platforms, analyzes mi… | 75 | 42341 | active |
| NewPipe NewPipe is a free, open-source, lightweight streaming front-end for Android that provides a privacy-friendly alternative to the official Yo… | 95 | 39818 | active |
| freeCodeCamp/devdocs DevDocs is an API documentation browser that combines multiple developer documentations into a single clean web UI with instant search, off… | 77 | 39546 | active |
| soxoj/maigret Maigret is a Python‑based command‑line tool that gathers public information about a person using only a username. It checks thousands of we… | 97 | 38084 | active |
| Lightpanda Browser Lightpanda Browser is a lightweight, headless web browser built from scratch in Zig, designed for AI agents and automation tasks. It offers… | 99 | 36755 | active |
| Lightpanda Browser Lightpanda is a headless web browser built from scratch in Zig, designed for AI agents, automation, and web scraping rather than human rend… | 99 | 36755 | active |
| changedetection.io changedetection.io is a self‑hosted web application that monitors web pages for content changes and sends alerts via many channels. It incl… | 94 | 34909 | active |
| changedetection.io A self-hosted web application for monitoring websites and detecting content changes, with alerts via Discord, Slack, Telegram, email, webho… | 94 | 34909 | active |
| AIHawk AIHawk is an open‑source AI browser agent that runs a stealthy, undetectable browser for web automation and scraping. It provides an MCP se… | 87 | 32150 | active |
| AIHawk AIHawk is an open-source Python AI agent that automates job applications by scraping job postings via browser automation and auto-applying … | 87 | 32150 | active |
| FujiwaraChoki/MoneyPrinterV2 MoneyPrinterV2 is a Python application that automates various online money-making workflows, including Twitter bots, YouTube Shorts automat… | 69 | 32101 | active |
| CloakBrowser CloakBrowser is a stealth Chromium binary with source‑level fingerprint patches that evades bot detection systems. It provides a drop‑in re… | 82 | 32001 | active |
| CloakBrowser CloakBrowser is a stealth Chromium distribution with 73 source-level C++ fingerprint patches that passes major bot-detection systems like C… | 82 | 32001 | active |
| iawia002/lux Lux is a fast and simple video downloader written in Go, available both as a CLI tool and a library. It supports downloading videos from ma… | 52 | 31775 | active |
| ScrapeGraphAI ScrapeGraphAI is a Python library that leverages large language models to automate web scraping and data extraction. It provides a high‑lev… | 87 | 31358 | active |
| ScrapeGraphAI ScrapeGraphAI is a Python library that uses LLMs and direct graph logic to build web scraping pipelines for websites and local documents (X… | 87 | 31358 | active |
| OpenCLI OpenCLI is a Node.js CLI tool that converts websites, browser sessions, and Electron apps into deterministic command-line interfaces for hu… | 79 | 30380 | active |
| GPT Researcher GPT Researcher is an open-source autonomous deep research agent that plans research questions, scrapes web and local sources, and produces … | 95 | 29943 | active |
| h4ckf0r0day/obscura Obscura is an open‑source headless browser engine written in Rust that speaks the Chrome DevTools Protocol. It provides isolated, fast‑star… | 82 | 28938 | active |
| ArchiveBox ArchiveBox is an open-source, self-hosted web archiving application that saves URLs, browser history, bookmarks, RSS feeds, and social medi… | 99 | 28737 | active |
| firecrawl/open-lovable Open Lovable is an AI-powered web application that lets users chat with an LLM to clone and recreate any website as a modern React app. It … | 63 | 28681 | active |
| xiaojieonly/Ehviewer_CN_SXJ EhViewer is an Android application for browsing and downloading content from the E-Hentai gallery. It provides a mobile-friendly interface … | 98 | 27320 | active |
| spotDL/spotify-downloader spotDL is a Python command-line tool that finds songs from Spotify playlists on YouTube and downloads them as MP3 files with album art, lyr… | 93 | 26209 | active |
| Crawlee Crawlee is an open‑source web scraping and browser automation library for JavaScript/TypeScript (and Python) that abstracts away headless b… | 98 | 26010 | active |
| Stagehand Stagehand is an SDK that lets developers control a browser with familiar Playwright‑style APIs and AI‑driven primitives. It provides self‑h… | 86 | 25583 | active |
| gocolly/colly Colly is a Go library that provides a clean, fast API for building web scrapers and crawlers. It supports synchronous, asynchronous, and pa… | 67 | 25552 | active |
| Max-Eee/NeoPass NeoPass is a Chrome extension that provides AI‑powered assistance and bypass tools for online exam portals such as Iamneo, HackerRank and N… | 86 | 24502 | active |
| qeeqbox/social-analyzer Social Analyzer is an OSINT tool available as a Web App, CLI, and API that finds and analyzes a person's profile across 1000+ social media … | 57 | 24114 | active |
| jhao104/proxy_pool A Python proxy pool service that periodically scrapes free proxies from 15+ sources, validates them, and stores them in Redis/SSDB. It expo… | 58 | 23762 | active |
| Mihon Mihon is a free and open source manga reader app for Android, a continuation of the Tachiyomi project. It lets users discover, download, an… | 88 | 23723 | active |
| Guzzle Guzzle is an extensible PHP HTTP client for sending HTTP requests and integrating with web services. It supports synchronous and asynchrono… | 98 | 23445 | stable |
| AZeC4/TelegramGroup A web service that aggregates and indexes over 10,000 Telegram groups, channels, and bots. It provides searchable listings and links to var… | 76 | 23431 | active |
| Skyvern-AI/skyvern Skyvern is an open‑source platform that adds AI and computer‑vision capabilities to Playwright for automating browser‑based workflows. It p… | 89 | 23226 | active |
| smicallef/spiderfoot SpiderFoot is an open-source OSINT automation tool that integrates with over 200 data sources to gather and analyze intelligence about targ… | 54 | 22608 | active |
| SingleFile SingleFile is an open-source browser extension for Chrome, Edge, Firefox, and Safari that saves an entire web page - including images, styl… | 99 | 22606 | stable |
| Neko Neko is a self-hosted virtual browser that runs in Docker and streams a fully functional browser desktop to multiple users over WebRTC with… | 95 | 22529 | active |
| browser-use/jev-ultrafast Jev Ultrafast is a Python-based browser agent that uses a small LLM to decide actions on web pages. It presents a dynamic, indexed action s… | 56 | 22407 | active |
| Builder.io A Node.js/TypeScript crawler that scrapes one or more websites and generates knowledge files (JSON) for creating custom GPTs or OpenAI assi… | 28 | 22405 | active |
| cat-catch Cat-Catch is an open‑source browser extension that lists and allows downloading of resources on the current page, including video streams. … | 90 | 22129 | active |
| cat-catch Cat-catch is an open-source browser extension that sniffs and lists media resources (video, audio, images) loaded by the current web page. … | 90 | 22129 | active |
| FreeTube FreeTube is a desktop YouTube client built with Electron and Vue that lets you watch videos without ads or tracking. It stores all user dat… | 86 | 22073 | active |
| jsdom jsdom is a pure‑JavaScript implementation of the DOM, HTML, and related web standards for Node.js. It lets you create and manipulate a virt… | 95 | 21716 | active |
| jsdom jsdom is a pure-JavaScript implementation of web standards like the WHATWG DOM and HTML Standards for Node.js. It emulates enough of a web … | 95 | 21716 | active |
| AutomaApp/automa Automa is a browser extension for Chrome and Firefox that lets users automate browser tasks by visually connecting blocks into workflows. I… | 62 | 21649 | active |
| Douyin_TikTok_Download_API Douyin_TikTok_Download_API is a self‑hosted, async REST API that scrapes Douyin and TikTok content and provides no‑watermark video download… | 98 | 20610 | active |
| Douyin_TikTok_Download_API A high-performance asynchronous Python service that scrapes and parses data from Douyin, TikTok, Kuaishou, and Bilibili, exposing a FastAPI… | 98 | 20610 | active |
| res-downloader Res‑Downloader is a cross‑platform desktop GUI built with Go and Wails that captures network traffic and extracts downloadable media resour… | 80 | 20510 | active |
| res-downloader A cross-platform desktop resource downloader built with Go and Wails that sniffs network traffic through a local proxy to capture videos, a… | 80 | 20510 | active |
| Obtainium Obtainium is an Android application that checks for updates directly from source repositories and downloads the latest APKs. It supports ma… | 95 | 20509 | active |
| Linkwarden Linkwarden is a self-hosted, open-source collaborative bookmark manager for collecting, reading, annotating, and preserving webpages. It ar… | 98 | 20155 | active |
| SETA OWL is a multi-agent collaboration framework built on the CAMEL-AI framework for automating real-world tasks through dynamic agent interact… | 65 | 20153 | active |
| Jack-Cherish/python-spider A Python3 collection of web‑scraping scripts and tutorials covering sites like Taobao, JD, Bilibili, 12306, Douyin, and more. It provides r… | 32 | 19805 | active |
| mikf/gallery-dl gallery-dl is a cross‑platform command‑line utility for downloading image galleries and collections from many popular hosting sites. It off… | 96 | 19781 | active |
| dzhng/deep-research An AI-powered deep research agent that iteratively generates search queries, scrapes web content, and refines its research direction using … | 52 | 19779 | active |
| segment-boneyard/nightmare Nightmare is a high-level browser automation library for JavaScript. It provides a simple API to control Chromium or Electron for tasks lik… | 23 | 19772 | active |
| lexmount/moli Moli is a production‑ready headless browser written in Rust, optimized for AI agents and lightweight web automation. It can be accessed via… | 81 | 19690 | active |
| browser-use/browser-harness Browser Harness is a self-healing Python harness that connects LLM agents directly to a real browser via a single editable CDP websocket. A… | 79 | 18360 | active |
| AgriciDaniel/claude-seo An open-source SEO skill/plugin for Claude Code that bundles 25 sub-skills, 18 specialist agents, and ~30 slash commands covering technical… | 82 | 18326 | active |
| laramies/theHarvester theHarvester is a Python CLI tool that gathers open-source intelligence (emails, subdomains, hostnames, IPs, names, URLs, ASNs) about a dom… | 89 | 18003 | active |
| ahmedkhaleel2004/gitdiagram GitDiagram is a web application that turns any public or private GitHub repository into an interactive architecture diagram in seconds, wit… | 66 | 17880 | active |
| getmaxun/maxun Maxun is an open-source no-code platform for web scraping, crawling, search, and AI-powered data extraction that turns websites into struct… | 94 | 17844 | active |
| projectdiscovery/katana Katana is a fast, configurable web crawling and spidering framework written in Go, supporting both standard HTTP-based and headless browser… | 90 | 17565 | active |
| citrolabs/ego-lite ego (lite) is a Chromium-based macOS browser designed to be shared between a human user and AI coding agents like Claude Code, Codex, and C… | 80 | 17025 | active |
| ffuf ffuf is a fast web fuzzer written in Go used for discovering directories, virtual hosts, and parameters by brute-forcing HTTP requests. It … | 96 | 16880 | stable |
| MODSetter/SurfSense SurfSense is an open-source, self-hostable NotebookLM alternative that combines a personal knowledge base with live open-web research conne… | 91 | 16505 | active |
| FreshRSS FreshRSS is a free, self-hosted RSS and Atom feed aggregator written in PHP, supporting multiple users, an anonymous reading mode, and a mo… | 95 | 16281 | stable |
| Xiaohongshu MCP A Model Context Protocol (MCP) server written in Go that lets AI assistants like Claude Code access and operate Xiaohongshu (Little Red Boo… | 79 | 16096 | active |
| avgupta456/github-trends GitHub Trends is a service that analyzes your GitHub commits via the GitHub API to compute lines-of-code statistics by language, repository… | 10 | 15954 | active |
| FlareSolverr FlareSolverr is a proxy server that bypasses Cloudflare and DDoS-GUARD protection by solving browser challenges with Selenium and undetecte… | 97 | 15906 | active |
| coderamp-labs/gitingest Gitingest turns any Git repository into a prompt-friendly text digest of its codebase, suitable for feeding into LLMs. It is available as a… | 68 | 15879 | active |
| Zotero Zotero is a free, open-source reference management application for collecting, organizing, annotating, citing, and sharing research sources… | 76 | 15465 | stable |
| dreammis/social-auto-upload A Python automation tool that uploads videos (and some image posts) to multiple social media platforms including Douyin, Bilibili, Xiaohong… | 59 | 15419 | active |
| Playwright for Python Playwright for Python is a library to automate Chromium, Firefox, and WebKit browsers with a single API, supporting both sync and async Pyt… | 92 | 15012 | stable |
| PuerkitoBio/goquery goquery is a Go library that provides a jQuery-like syntax for parsing, traversing, and manipulating HTML documents, built on Go's net/html… | 87 | 14996 | stable |
| Holehe Holehe is an OSINT tool that checks whether an email address is registered on 120+ sites (Twitter, Instagram, Snapchat, etc.) by abusing th… | 32 | 14958 | active |
| Datalux/Osintgram Osintgram is a Python-based OSINT tool for Instagram that provides an interactive shell to collect and analyze information about Instagram … | 86 | 14936 | active |
| alexta69/metube MeTube is a self-hosted web UI for yt-dlp that lets users download videos, audio, playlists, and channels from YouTube and dozens of other … | 93 | 14932 | active |
| maurosoria/dirsearch dirsearch is an advanced web path scanner that brute-forces directories and files on web servers using wordlists. It is a Python CLI tool w… | 84 | 14811 | active |
| Usagi-org/ai-goofish-monitor A self-hosted Python application that monitors Xianyu (Goofish, China's second-hand marketplace) listings in real time or on cron schedules… | 66 | 14612 | active |
| Radarr/Radarr Radarr is a self-hosted movie collection manager for Usenet and BitTorrent users that monitors RSS feeds, searches indexers, and sends rele… | 98 | 14560 | active |
| OJ/gobuster Gobuster is a fast, multi-threaded brute-forcing tool written in Go for enumerating web directories/files, DNS subdomains, virtual hosts, c… | 76 | 14150 | active |
| SponsorBlock SponsorBlock is an open-source crowdsourced browser extension that automatically skips sponsor segments and other annoying parts (intros, o… | 94 | 13890 | active |
page 1 / 20 next →