function: web-scraping
1985 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| Firecrawl Firecrawl is an open-source web scraping and crawling API that turns websites into clean Markdown, structured JSON, screenshots, and other … | 88 | 172809 | active |
| Browser Use Browser Use is a Python library that lets LLM-powered AI agents control a web browser like a human — opening pages, clicking, typing, and f… | 84 | 110969 | active |
| microsoft/playwright Playwright is a framework for web testing and browser automation that drives Chromium, Firefox, and WebKit through a single API, available … | 98 | 95186 | stable |
| unclecode/crawl4ai Crawl4AI is an open-source Python library that crawls websites with a headless browser and converts pages into clean, LLM-ready Markdown fo… | 89 | 79471 | active |
| D4Vinci/Scrapling Scrapling is an adaptive Python web scraping framework that handles everything from single requests to full-scale concurrent crawls. It fea… | 89 | 76660 | active |
| Panniantong/Agent-Reach Agent Reach is a Python CLI and MCP tool that gives AI agents the ability to read and search across platforms like Twitter, Reddit, YouTube… | 78 | 75643 | active |
| scrapy/scrapy Scrapy is a fast, high-level web crawling and scraping framework for Python used to extract structured data from websites. It provides a fu… | 99 | 64048 | stable |
| NanmiCoder/MediaCrawler MediaCrawler is a Python-based multi-platform social media crawler that scrapes notes, videos, posts, and comments from Xiaohongshu, Douyin… | 73 | 63826 | active |
| FoundationAgents/OpenManus OpenManus is an open-source framework for building general-purpose AI agents, created by the MetaGPT team as an open alternative to Manus. … | 66 | 58085 | active |
| soimort/you-get You-Get is a tiny command-line utility written in Python to download media content (videos, audios, images) from the web when no other hand… | 67 | 56871 | active |
| huginn/huginn Huginn is a self-hosted system for building agents that perform automated tasks online, such as monitoring websites, watching Twitter, and … | 67 | 49857 | active |
| ChromeDevTools/chrome-devtools-mcp An MCP server that exposes Chrome DevTools capabilities to AI coding agents, letting them control and inspect a live Chrome browser via Pup… | 84 | 49759 | active |
| DIYgod/RSSHub RSSHub is an open-source RSS feed generator that turns websites, social media platforms, and apps (Twitter, YouTube, Telegram, Bilibili, et… | 67 | 45898 | active |
| NaiboWang/EasySpider EasySpider is a free, open-source visual no-code web crawler and browser automation (RPA) tool where users design scraping tasks by clickin… | 87 | 44442 | active |
| imputnet/cobalt Cobalt is a friendly, ad-free media downloader web app that lets users save video, audio, photos, and GIFs from sites like YouTube, Twitter… | 66 | 42273 | active |
| 666ghj/BettaFish BettaFish (微舆) is a from-scratch multi-agent public opinion analysis system in Python that monitors 30+ social media platforms, analyzes mi… | 77 | 42090 | active |
| NewPipe NewPipe is a free, open-source, lightweight streaming front-end for Android that provides a privacy-friendly alternative to the official Yo… | 99 | 39500 | active |
| freeCodeCamp/devdocs DevDocs is an API documentation browser that combines multiple developer documentations into a single clean web UI with instant search, off… | 77 | 39378 | active |
| soxoj/maigret Maigret is a Python CLI OSINT tool that builds a dossier on a person from just a username, checking thousands of sites for accounts and ext… | 99 | 37084 | active |
| Selenium Selenium is an umbrella project of tools and libraries for automating web browsers, providing the infrastructure for the W3C WebDriver spec… | 94 | 34397 | stable |
| lightpanda-io/browser Lightpanda is a headless web browser built from scratch in Zig, designed for AI agents, automation, and web scraping rather than human rend… | 97 | 34269 | active |
| dgtlmoon/changedetection.io A self-hosted web application for monitoring websites and detecting content changes, with alerts via Discord, Slack, Telegram, email, webho… | 93 | 33356 | active |
| FujiwaraChoki/MoneyPrinterV2 MoneyPrinterV2 is a Python application that automates various online money-making workflows, including Twitter bots, YouTube Shorts automat… | 65 | 31675 | active |
| iawia002/lux Lux is a fast and simple video downloader written in Go, available both as a CLI tool and a library. It supports downloading videos from ma… | 56 | 31656 | active |
| CloakHQ/CloakBrowser CloakBrowser is a stealth Chromium distribution with 73 source-level C++ fingerprint patches that passes major bot-detection systems like C… | 81 | 30849 | active |
| cheeriojs/cheerio Cheerio is a fast, flexible JavaScript library for parsing and manipulating HTML and XML with a jQuery-like API. It works in both browser a… | 84 | 30468 | stable |
| chubin/wttr.in wttr.in is a console-oriented weather forecast service that renders weather reports as ANSI terminal art, HTML, or PNG, accessible via curl… | 75 | 30415 | active |
| feder-cr/Jobs_Applier_AI_Agent_AIHawk AIHawk is an open-source Python AI agent that automates job applications by scraping job postings via browser automation and auto-applying … | 58 | 30260 | active |
| ScrapeGraphAI/Scrapegraph-ai ScrapeGraphAI is a Python library that uses LLMs and direct graph logic to build web scraping pipelines for websites and local documents (X… | 88 | 29959 | active |
| Kazumi Kazumi is a cross-platform anime streaming and aggregation app built with Flutter that uses user-defined XPath-based rules to scrape and wa… | 87 | 29257 | active |
| assafelovic/gpt-researcher GPT Researcher is an open-source autonomous deep research agent that plans research questions, scrapes web and local sources, and produces … | 96 | 29160 | active |
| jackwener/OpenCLI OpenCLI is a Node.js CLI tool that converts websites, browser sessions, and Electron apps into deterministic command-line interfaces for hu… | 77 | 28619 | active |
| firecrawl/open-lovable Open Lovable is an AI-powered web application that lets users chat with an LLM to clone and recreate any website as a modern React app. It … | 42 | 28327 | active |
| ArchiveBox/ArchiveBox ArchiveBox is an open-source, self-hosted web archiving application that saves URLs, browser history, bookmarks, RSS feeds, and social medi… | 83 | 28188 | active |
| spotDL/spotify-downloader spotDL is a Python command-line tool that finds songs from Spotify playlists on YouTube and downloads them as MP3 files with album art, lyr… | 96 | 25921 | active |
| Crawlee Crawlee is a web scraping and browser automation library for Node.js/TypeScript (with a Python port) for building reliable crawlers. It int… | 98 | 25516 | stable |
| gocolly/colly Colly is a fast and elegant web scraping and crawling framework for Go. It provides a clean callback-based API for making HTTP requests, pa… | 66 | 25483 | stable |
| Guovin/iptv-api An automated IPTV live-stream source updater that aggregates channels from multiple sources, validates availability, speed-tests them, and … | 92 | 25003 | active |
| Stagehand Stagehand is an SDK for building browser agents, offering Playwright-style APIs combined with AI primitives (act, extract, observe) that us… | 87 | 24069 | active |
| qeeqbox/social-analyzer Social Analyzer is an OSINT tool available as a Web App, CLI, and API that finds and analyzes a person's profile across 1000+ social media … | 60 | 23835 | active |
| jhao104/proxy_pool A Python proxy pool service that periodically scrapes free proxies from 15+ sources, validates them, and stores them in Redis/SSDB. It expo… | 62 | 23642 | active |
| guzzle/guzzle Guzzle is an extensible PHP HTTP client for sending HTTP requests and integrating with web services. It supports synchronous and asynchrono… | 99 | 23457 | stable |
| mihonapp/mihon Mihon is a free and open source manga reader app for Android, a continuation of the Tachiyomi project. It lets users discover, download, an… | 92 | 23136 | active |
| Skyvern-AI/skyvern Skyvern is an open-source AI browser automation framework that uses LLMs and computer vision to interact with websites, offering a Playwrig… | 88 | 22852 | active |
| BuilderIO/gpt-crawler A Node.js/TypeScript crawler that scrapes one or more websites and generates knowledge files (JSON) for creating custom GPTs or OpenAI assi… | 31 | 22393 | active |
| h4ckf0r0day/obscura Obscura is an open-source headless browser engine written in Rust that runs JavaScript via V8 and speaks the Chrome DevTools Protocol, acti… | 81 | 22334 | active |
| SingleFile SingleFile is an open-source browser extension for Chrome, Edge, Firefox, and Safari that saves an entire web page - including images, styl… | 67 | 22251 | stable |
| AKShare AKShare is a Python library providing a unified interface for fetching financial and economic data (stocks, futures, bonds, options, funds,… | 95 | 22247 | active |
| m1k1o/neko Neko is a self-hosted virtual browser that runs in Docker and streams a fully functional browser desktop to multiple users over WebRTC with… | 90 | 22193 | active |
| Matomo Matomo (formerly Piwik) is the leading open-source web analytics platform, a privacy-focused alternative to Google Analytics built in PHP a… | 99 | 21807 | stable |
| jsdom/jsdom jsdom is a pure-JavaScript implementation of web standards like the WHATWG DOM and HTML Standards for Node.js. It emulates enough of a web … | 94 | 21659 | active |
| AutomaApp/automa Automa is a browser extension for Chrome and Firefox that lets users automate browser tasks by visually connecting blocks into workflows. I… | 66 | 21586 | active |
| xifangczy/cat-catch Cat-catch is an open-source browser extension that sniffs and lists media resources (video, audio, images) loaded by the current web page. … | 97 | 21548 | active |
| ourongxing/newsnow NewsNow is a self-hostable web application that aggregates real-time and trending news from multiple sources with a clean, elegant reading … | 82 | 21513 | active |
| smicallef/spiderfoot SpiderFoot is an open-source OSINT automation tool that integrates with over 200 data sources to gather and analyze intelligence about targ… | 57 | 21441 | active |
| camel-ai/owl OWL is a multi-agent collaboration framework built on the CAMEL-AI framework for automating real-world tasks through dynamic agent interact… | 64 | 20104 | active |
| Evil0ctal/Douyin_TikTok_Download_API A high-performance asynchronous Python service that scrapes and parses data from Douyin, TikTok, Kuaishou, and Bilibili, exposing a FastAPI… | 55 | 19679 | active |
| linkwarden/linkwarden Linkwarden is a self-hosted, open-source collaborative bookmark manager for collecting, reading, annotating, and preserving webpages. It ar… | 98 | 19606 | active |
| dzhng/deep-research An AI-powered deep research agent that iteratively generates search queries, scrapes web content, and refines its research direction using … | 55 | 19603 | active |
| putyy/res-downloader A cross-platform desktop resource downloader built with Go and Wails that sniffs network traffic through a local proxy to capture videos, a… | 81 | 19522 | active |
| mikf/gallery-dl gallery-dl is a command-line program to download image galleries and collections from many image hosting sites such as Pixiv, Danbooru, Dev… | 96 | 19332 | active |
| projectdiscovery/katana Katana is a fast, configurable web crawling and spidering framework written in Go, supporting both standard HTTP-based and headless browser… | 94 | 17348 | active |
| getmaxun/maxun Maxun is an open-source no-code platform for web scraping, crawling, search, and AI-powered data extraction that turns websites into struct… | 94 | 17299 | active |
| laramies/theHarvester theHarvester is a Python CLI tool that gathers open-source intelligence (emails, subdomains, hostnames, IPs, names, URLs, ASNs) about a dom… | 91 | 17202 | active |
| browser-use/browser-harness Browser Harness is a self-healing Python harness that connects LLM agents directly to a real browser via a single editable CDP websocket. A… | 81 | 17160 | active |
| ffuf/ffuf ffuf is a fast web fuzzer written in Go used for discovering directories, virtual hosts, and parameters by brute-forcing HTTP requests. It … | 97 | 16593 | stable |
| MODSetter/SurfSense SurfSense is an open-source, self-hostable NotebookLM alternative that combines a personal knowledge base with live open-web research conne… | 90 | 16017 | active |
| avgupta456/github-trends GitHub Trends is a service that analyzes your GitHub commits via the GitHub API to compute lines-of-code statistics by language, repository… | 10 | 15987 | active |
| ahmedkhaleel2004/gitdiagram GitDiagram is a web application that turns any public or private GitHub repository into an interactive architecture diagram in seconds, wit… | 65 | 15915 | active |
| FreshRSS/FreshRSS FreshRSS is a free, self-hosted RSS and Atom feed aggregator written in PHP, supporting multiple users, an anonymous reading mode, and a mo… | 90 | 15849 | stable |
| xpzouying/xiaohongshu-mcp A Model Context Protocol (MCP) server written in Go that lets AI assistants like Claude Code access and operate Xiaohongshu (Little Red Boo… | 80 | 15495 | active |
| AgriciDaniel/claude-seo An open-source SEO skill/plugin for Claude Code that bundles 25 sub-skills, 18 specialist agents, and ~30 slash commands covering technical… | 82 | 15372 | active |
| coderamp-labs/gitingest Gitingest turns any Git repository into a prompt-friendly text digest of its codebase, suitable for feeding into LLMs. It is available as a… | 68 | 15350 | active |
| FlareSolverr/FlareSolverr FlareSolverr is a proxy server that bypasses Cloudflare and DDoS-GUARD protection by solving browser challenges with Selenium and undetecte… | 95 | 15316 | active |
| Zotero Zotero is a free, open-source reference management application for collecting, organizing, annotating, citing, and sharing research sources… | 77 | 15077 | stable |
| PuerkitoBio/goquery goquery is a Go library that provides a jQuery-like syntax for parsing, traversing, and manipulating HTML documents, built on Go's net/html… | 86 | 14981 | stable |
| microsoft/playwright-python Playwright for Python is a library to automate Chromium, Firefox, and WebKit browsers with a single API, supporting both sync and async Pyt… | 93 | 14952 | stable |
| maurosoria/dirsearch dirsearch is an advanced web path scanner that brute-forces directories and files on web servers using wordlists. It is a Python CLI tool w… | 87 | 14662 | active |
| dreammis/social-auto-upload A Python automation tool that uploads videos (and some image posts) to multiple social media platforms including Douyin, Bilibili, Xiaohong… | 61 | 14594 | active |
| alexta69/metube MeTube is a self-hosted web UI for yt-dlp that lets users download videos, audio, playlists, and channels from YouTube and dozens of other … | 94 | 14522 | active |
| megadose/holehe Holehe is an OSINT tool that checks whether an email address is registered on 120+ sites (Twitter, Instagram, Snapchat, etc.) by abusing th… | 32 | 14278 | active |
| Usagi-org/ai-goofish-monitor A self-hosted Python application that monitors Xianyu (Goofish, China's second-hand marketplace) listings in real time or on cron schedules… | 10 | 14242 | active |
| Radarr/Radarr Radarr is a self-hosted movie collection manager for Usenet and BitTorrent users that monitors RSS feeds, searches indexers, and sends rele… | 97 | 14236 | active |
| Datalux/Osintgram Osintgram is a Python-based OSINT tool for Instagram that provides an interactive shell to collect and analyze information about Instagram … | 40 | 14147 | active |
| OJ/gobuster Gobuster is a fast, multi-threaded brute-forcing tool written in Go for enumerating web directories/files, DNS subdomains, virtual hosts, c… | 80 | 14038 | active |
| citrolabs/ego-lite ego (lite) is a Chromium-based macOS browser designed to be shared between a human user and AI coding agents like Claude Code, Codex, and C… | 80 | 13772 | active |
| SponsorBlock SponsorBlock is an open-source crowdsourced browser extension that automatically skips sponsor segments and other annoying parts (intros, o… | 97 | 13702 | active |
| pystardust/ani-cli ani-cli is a POSIX shell script CLI tool for browsing and watching anime from the terminal, scraping the anidb site and playing videos via … | 97 | 13602 | active |
| HackerNews/API Official documentation and samples for the Hacker News public API, hosted on Firebase. It describes how to access HN items (stories, commen… | 32 | 13276 | stable |
| chromedp chromedp is a Go library for driving Chrome and other browsers via the Chrome DevTools Protocol, with no external dependencies. It supports… | 84 | 13264 | active |
| instaloader/instaloader Instaloader is a Python command-line tool and library for downloading pictures, videos, captions, comments, and metadata from Instagram. It… | 87 | 13240 | active |
| mealie-recipes/mealie Mealie is a self-hosted recipe manager and meal planner with a FastAPI REST backend and a Vue frontend. It imports recipes from URLs via a … | 99 | 13079 | active |
| seleniumbase/SeleniumBase SeleniumBase is an all-in-one Python browser automation framework for web testing, crawling, and scraping, built on Selenium/WebDriver with… | 95 | 12955 | active |
| wallabag/wallabag wallabag is a self-hostable web application for saving web pages to read later, extracting article content while stripping ads and distract… | 78 | 12937 | stable |
| ultrafunkamsterdam/undetected-chromedriver A Python library that patches Selenium's ChromeDriver binary so automated Chrome sessions avoid detection by anti-bot systems like Cloudfla… | 46 | 12808 | active |
| The-Pocket/PocketFlow-Tutorial-Codebase-Knowledge An AI agent built with the Pocket Flow LLM framework that crawls GitHub repositories, analyzes codebases to identify core abstractions, and… | 58 | 12632 | active |
| elebumm/RedditVideoMakerBot A Python application that automatically generates Reddit story videos for platforms like TikTok, YouTube, and Instagram with a single comma… | 72 | 12532 | active |
| JoeanAmier/XHS-Downloader A tool for extracting links and downloading content (images, videos, live photos) from XiaoHongShu (RedNote) posts. It ships as a Python TU… | 76 | 12489 | active |
| g1879/DrissionPage DrissionPage is a Python-based web automation library that combines browser control (via Chromium's DevTools Protocol, without WebDriver) w… | 70 | 12405 | active |
| hangwin/mcp-chrome A Chrome extension that acts as a Model Context Protocol (MCP) server, exposing your real Chrome browser—its tabs, login states, and config… | 56 | 12352 | active |
page 1 / 20 next →