Open-source crawlers software
569 open-source libraries, tools and applications tagged crawlers, ranked by how closely crawlers is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
569 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| kotartemiy/pygooglenews A Python wrapper around the Google News RSS feed providing top stories, topic and geolocation feeds, and full-text search with date-range a… | 32 | 1396 | maintenance |
| PKUJohnson/OpenData OpenDataTools is a Python library that scrapes financial and investment data from various websites and exposes it through simple, easy-to-u… | 32 | 1369 | maintenance |
| martinsbalodis/web-scraper-chrome-extension Web Scraper is a Chrome browser extension for extracting data from web pages without writing code. Users define sitemaps describing how to … | 32 | 1363 | maintenance |
| huaying/instagram-crawler A Python command-line crawler that scrapes Instagram posts, profiles, and hashtag data using Selenium and ChromeDriver, without the officia… | 32 | 1362 | maintenance |
| felipecsl/wombat Wombat is a lightweight Ruby web crawler and scraper library with an elegant DSL for extracting structured data from web pages. It lets dev… | 67 | 1360 | maintenance |
| adriancooney/puppeteer-heap-snapshot A TypeScript library and CLI for capturing Chrome DevTools heap snapshots from Puppeteer-controlled pages and querying them for objects wit… | 23 | 1347 | maintenance |
| scrapinghub/frontera Frontera is a Python web crawling framework that implements a scalable crawl frontier, storing and prioritizing links extracted by crawlers… | 30 | 1334 | maintenance |
| laramies/metagoofil Metagoofil is a Python command-line OSINT tool that searches Google for public documents (pdf, doc, xls, ppt) on target websites, downloads… | 32 | 1333 | maintenance |
| devanshbatham/FavFreak A Python CLI tool that fetches favicon.ico files from lists of URLs, computes their mmh3 hashes, and groups domains/subdomains/IPs by match… | 23 | 1314 | maintenance |
| c4tcom/Katana Katana-ds is a Python CLI tool that automates advanced Google queries known as Google Dorks (Google Dorking), with optional Tor support for… | 10 | 1310 | maintenance |
| InstaPy/instagram-profilecrawl A Python script from the InstaPy ecosystem that crawls public Instagram profile information such as post counts, follower counts, and post … | 32 | 1308 | maintenance |
| Sniper970119/dianping_spider A Python web scraper for Dianping (大众点评) that extracts search results, shop details, and reviews, handling dynamic font encryption without … | 69 | 1295 | maintenance |
| sunra/php-simple-html-dom-parser A PHP library adapting Simple HTML DOM Parser for Composer and PSR-0, allowing easy HTML parsing and manipulation. It supports invalid HTML… | 23 | 1283 | maintenance |
| useragents/Zefoy-TikTok-Automator A Python/Selenium automation script that drives zefoy.com to send TikTok followers, views, likes, shares, favorites, and comment likes. It … | 32 | 1278 | maintenance |
| dragnet-org/dragnet Dragnet is a Python library that uses machine learning models to extract the main article content, and optionally user comments, from HTML … | 33 | 1277 | maintenance |
| Beanbun Beanbun is a multi-process web crawler framework written in PHP, built on Workerman with Guzzle as the default downloader. It supports dist… | 57 | 1261 | maintenance |
| dchrastil/ScrapedIn A Python CLI tool that scrapes LinkedIn without API restrictions to enumerate employees of a target company for red team or social engineer… | 32 | 1241 | maintenance |
| istresearch/scrapy-cluster Scrapy Cluster is a distributed web scraping framework built on Scrapy that uses Redis to coordinate crawl requests and Kafka as a data bus… | 10 | 1222 | maintenance |
| vysecurity/LinkedInt LinkedInt is a Python CLI tool for LinkedIn reconnaissance that scrapes employee profiles for a target company and generates an HTML report… | 10 | 1214 | maintenance |
| Mapaler/PixivUserBatchDownload A userscript (browser extension-style script) that batch-downloads all public works of a Pixiv artist directly from the Pixiv website, send… | 44 | 1198 | maintenance |
| bonfy/github-trending A Python script that scrapes GitHub's trending page daily and archives the most popular repositories as Markdown files. It is designed to r… | 76 | 1189 | maintenance |
| timwhitez/crawlergo_x_XRAY A Python glue script that combines the crawlergo dynamic crawler with the XRAY passive vulnerability scanner, replaying crawled URLs throug… | 32 | 1182 | maintenance |
| yutto-dev/bilili bilili is a Python CLI tool for downloading Bilibili videos (including bangumi/season content) along with danmaku comments and subtitles. I… | 10 | 1175 | maintenance |
| WebSpiderUtils/verification_code A research repository documenting approaches and code for solving mainstream CAPTCHA systems such as Geetest, NetEase Yidun, and Aliyun CAP… | 32 | 1168 | maintenance |
| dixudx/tumblr-crawler A Python script that downloads all photos and videos from specified Tumblr blogs. It supports batch site lists, proxy configuration, and sk… | 55 | 1160 | maintenance |
| holgerd77/django-dynamic-scraper A Django app that builds on the Scrapy framework and lets you create and manage Scrapy spiders through the Django admin interface. It remov… | 23 | 1154 | maintenance |
| s045pd/DarkNet_ChineseTrading A Python-based real-time crawler that monitors Chinese-language darknet marketplaces over Tor, with automatic account registration, login, … | 10 | 1151 | maintenance |
| Jinnrry/RobotHelper RobotHelper is an Android automation script framework written in Java, providing common building blocks like screen capture, image-based po… | 23 | 1145 | maintenance |
| juancarlospaco/faster-than-requests A Python 3 HTTP client library claiming to be much faster than the popular Requests library, implemented with a compiled core and minimal d… | 67 | 1127 | maintenance |
| kohlschutter/boilerpipe boilerpipe is a Java library for removing boilerplate (ads, navigation, headers) from HTML pages and extracting the main full text content.… | 32 | 1127 | maintenance |
| medialab/artoo artoo.js is a JavaScript library injected into a webpage's context (typically via a bookmarklet) that provides client-side web scraping uti… | 23 | 1117 | maintenance |
| exorde-labs/exorde-client The Exorde client is a Python CLI worker node for the Exorde Network, a decentralized protocol where participants scrape social media and w… | 54 | 1081 | maintenance |
| acikyazilimagi/afet-org An open-source earthquake relief platform (depremyardim.com / afetharita.com) that aggregates calls for help from Twitter, WhatsApp, Telegr… | 31 | 1077 | maintenance |
| jonbakerfish/TweetScraper TweetScraper is a Scrapy-based crawler that scrapes tweets and user information from Twitter Search without using Twitter's official APIs. … | 23 | 1063 | maintenance |
| utkarshkukreti/select.rs A Rust library for extracting useful data from HTML documents, built around predicate-based DOM queries. It is designed for web scraping ta… | 34 | 1017 | maintenance |
| Algebra-FUN/WeReadScan A Python library that uses Selenium headless browsers to scan purchased books from WeRead (WeChat Reading) and convert them into local PDF … | 32 | 1003 | maintenance |
| gsh199449/spider Gather Platform is a Java-based configurable web crawler built on the WebMagic kernel, with a web console for configuring and managing scra… | 57 | 997 | maintenance |
| JSREI/ast-hook-for-js-RE A browser memory roaming tool for JavaScript reverse engineering that hooks variable assignments via AST-transformed proxy responses. It le… | 23 | 1911 | experimental |
| tinyfish-io/bigset-oss BigSet is a self-hostable application that turns a natural-language sentence into a structured, regularly refreshed dataset by dispatching … | 74 | 1711 | experimental |
| kkyon/botflow Botflow is a Python dataflow programming framework for building data pipelines using pipes and routes, with parallelism via coroutines and … | 58 | 1193 | experimental |
| oxylabs/ai-scraper-py AI-Scraper is a Python library and scrape agent from Oxylabs AI Studio that extracts data from webpages using natural language prompts inst… | 10 | 1090 | experimental |
| twintproject/twint Twint is a Python CLI tool and library that scrapes tweets, followers, following, and likes from Twitter without using the official API or … | 10 | 16401 | abandoned |
| wechat-article/wechat-article-exporter An online batch downloader for WeChat Official Account articles that exports posts in HTML, JSON, Excel, TXT, Markdown, and DOCX formats, i… | 69 | 13012 | abandoned |
| xiandanin/magnetW MagnetW is a cross-platform desktop application built with Electron and Vue that aggregates magnet link search results from multiple torren… | 10 | 11262 | abandoned |
| yangyangwithgnu/hardseed hardseed is a C++ command-line tool that scrapes adult forum threads (aicheng, caoliu) to batch-download images and torrent seed files, wit… | 32 | 9196 | abandoned |
| TonyChen56/WeChatRobot A C++ WeChat hooking toolkit and robot framework providing wxhook APIs, WeChat database decryption, and Official Account (公众号) scraping, pl… | 54 | 7217 | abandoned |
| xchaoinfo/fuck-login A Python library that simulates programmatic login to popular Chinese websites like Zhihu, Weibo, Baidu, and Douban, built on requests, Pil… | 10 | 5868 | abandoned |
| hanc00l/wooyun_public A crawler and search application for the archived Wooyun.org security vulnerability disclosure platform, containing ~40k-88k public vulnera… | 10 | 4392 | abandoned |
| qiyeboy/IPProxyPool IPProxyPool is a Python proxy pool service that crawls free proxy IPs from the web, validates them, stores them in a database (SQLite by de… | 23 | 4288 | abandoned |
| bisguzar/twitter-scraper A Python library that scrapes Twitter's frontend JavaScript API without authentication, letting users fetch tweets from profiles or hashtag… | 10 | 4004 | abandoned |
| pyppeteer/pyppeteer Pyppeteer is an unofficial Python port of Puppeteer for automating headless Chrome/Chromium browsers via asyncio. It supports page navigati… | 32 | 3943 | abandoned |
| miyakogi/pyppeteer An unofficial Python port of Puppeteer for automating headless Chrome/Chromium browsers via asyncio. This original repository has moved to … | 10 | 3551 | abandoned |
| bowenpay/wechat-spider A Python-based web crawler for scraping articles from WeChat public accounts (微信公众号), built on Django with MySQL and Redis, including a web… | 32 | 3385 | abandoned |
| LiuXingMing/SinaSpider A Python web crawler for Sina Weibo (Chinese microblog) built on Scrapy, with three versions: a standalone spider, a distributed version us… | 32 | 3283 | abandoned |
| gnemoug/distribute_crawler A distributed web crawler built on Scrapy, Redis, MongoDB, and Graphite, demonstrated with a spider for a Chinese book-download site. Redis… | 32 | 3234 | abandoned |
| harismuneer/Ultimate-Social-Scrapers A collection of Python-based scraping tools that extract public data from Facebook, Instagram, and Twitter (X), including posts, media, fol… | 40 | 3176 | abandoned |
| airingursb/bilibili-user A Python web crawler that scrapes Bilibili user profiles (id, nickname, gender, avatar, level, birthday, location, etc.) and stores them in… | 32 | 3090 | abandoned |
| NikolaiT/GoogleScraper GoogleScraper is a Python module and CLI tool for scraping search engine results from Google, Bing, Yandex, DuckDuckGo and others, with sup… | 32 | 2886 | abandoned |
| YahooArchive/anthelion Anthelion is an Apache Nutch plugin for focused crawling of semantic data embedded in HTML pages. It uses an online learning classifier to … | 10 | 2823 | abandoned |
| lanbing510/DouBanSpider A Python web scraper for Douban Books that crawls book listings by tag, storing ratings and review counts into Excel files. The author also… | 32 | 2787 | abandoned |
| loadchange/amemv-crawler A Python 3 script that downloads all videos from a specified Douyin (TikTok China) user account, as well as all videos under a given challe… | 32 | 2660 | abandoned |
| taspinar/twitterscraper A Python library that scrapes tweets and user information from Twitter using requests and BeautifulSoup, without relying on Twitter's offic… | 23 | 2460 | abandoned |
| scrapoxy/scrapoxy Scrapoxy was an open-source proxy manager for web scraping that aggregated proxies from cloud providers and other sources behind a single A… | 58 | 2410 | abandoned |
| egrcc/zhihu-python A Python 2.7 library for scraping content from Zhihu, a Chinese Q&A platform, including questions, answers, users, and favorites. It can ex… | 32 | 2336 | abandoned |
| chiphuyen/lazynlp A Python library for crawling, cleaning, and deduplicating web pages to build massive monolingual text datasets, suitable for training lang… | 23 | 2298 | abandoned |
| PaulMcInnis/JobFunnel JobFunnel is a Python CLI tool that scrapes job postings from multiple job websites (Indeed, Glassdoor, LinkedIn) into a single deduplicate… | 10 | 2189 | abandoned |
| minimaxir/facebook-page-post-scraper A Python script collection that scrapes all posts, reactions, and comments from public Facebook Pages and open Groups via the Facebook Grap… | 10 | 2134 | abandoned |
| simplecrawler/simplecrawler simplecrawler is a flexible, event-driven web crawler library for Node.js with a configurable queue system, robots.txt support, and link di… | 10 | 2130 | abandoned |
| althonos/InstaLooter InstaLooter is a Python CLI tool that downloads pictures and videos from Instagram profiles without using the official API. It is a re-impl… | 23 | 2106 | abandoned |
| cycz/jdBuyMask A Python automation tool that monitored JD.com (Jingdong) for face mask stock during the COVID-19 pandemic and automatically placed purchas… | 32 | 1848 | abandoned |
| hu17889/go_spider go_spider is a concurrent web crawler framework written in Go, designed for crawling vertical communities with a flexible, modular architec… | 23 | 1817 | abandoned |
| node-js-libs/node.io node.io is a Node.js web scraping and data extraction library originally written in 2010. It is explicitly no longer maintained, with the a… | 32 | 1790 | abandoned |
| bughandler/cnki-downloader A small desktop tool for searching and downloading academic literature from CNKI (China National Knowledge Infrastructure). Its backend int… | 44 | 1766 | abandoned |
| ZFC-Digital/puppeteer-real-browser A Node.js library that wraps Puppeteer with a real-browser profile to bypass bot detection systems like Cloudflare and Turnstile captchas. … | 32 | 1648 | abandoned |
| erma0/douyin A Python crawler for Douyin (Chinese TikTok) that collected public data such as account profiles, likes, favorites, music, hashtags, search… | 72 | 1616 | abandoned |
| fossasia/loklak_wok_android Loklak Wok is an Android app that acts as a harvesting peer for the loklak_server, collecting social media messages (tweets) and pushing th… | 10 | 1558 | abandoned |
| GravityLabs/goose Goose is a Scala library (originally Java) that extracts the main body text, metadata, publish date, embedded videos, and top image from ne… | 10 | 1524 | abandoned |
| yhat/scrape A Go library providing a higher-level interface over golang.org/x/net/html for web scraping. It offers generic tree traversal helpers like … | 10 | 1513 | abandoned |
| qinxuye/cola Cola is a high-level distributed crawling framework in Python for scraping pages and extracting structured data from websites. The same cra… | 10 | 1499 | abandoned |
| johntitus/node-horseman A Node.js library providing a chainable, Promise-based API for controlling the PhantomJS headless browser, supporting page navigation, form… | 32 | 1480 | abandoned |
| fossasia/loklak_scraper_js A collection of JavaScript scrapers for the loklak project that extract data from websites like Twitter and Quora and output JSON resemblin… | 10 | 1471 | abandoned |
| keenwon/antcolony AntColony is a Node.js-based BitTorrent DHT network crawler that collects active infohashes, downloads and parses torrent files, and stores… | 32 | 1454 | abandoned |
| jamesturk/scrapeghost scrapeghost is an experimental Python library that uses OpenAI's GPT models to scrape structured data from websites without writing page-sp… | 46 | 1439 | abandoned |
| OpnTec/parliament-scraper A collection of scrapers (in Python, Ruby, and Scala) that download public parliamentary data such as written questions from the EU Parliam… | 32 | 1404 | abandoned |
| leonardocardoso/SwiftLinkPreview A Swift library that generates link previews from URLs by extracting titles, relevant text, and images. It supports iOS, macOS, watchOS, an… | 10 | 1385 | abandoned |
| dinubs/jam-api Jam API is a hosted service (self-hostable Node.js app) that turns any website into a JSON API by extracting data using CSS query selectors… | 32 | 1359 | abandoned |
| Vespa314/bilibili-api A collection of Bilibili (B站) API documentation and Python tooling for scraping video, user, comment, danmaku, and bangumi data. It include… | 32 | 1357 | abandoned |
| Jefferson-Henrique/GetOldTweets-python A Python library that retrieves old tweets by mimicking the JSON calls Twitter Search makes in the browser, bypassing the official API's ti… | 32 | 1339 | abandoned |
| JD_AutoBuy A Python 2.7 command-line scraper that logs into JD.com (via QR code or credentials), monitors product stock and price, and automatically p… | 32 | 1309 | abandoned |
| LiuRoy/zhihu_spider A Scrapy-based web crawler that scrapes Zhihu user profiles and their follower/following relationship graphs, storing data in MongoDB. It s… | 32 | 1289 | abandoned |
| MiningCattiva/x-spider X-Spider is a desktop media downloader for X (Twitter) that fetches images and videos from accounts. It supports media filters, configurabl… | 10 | 1258 | abandoned |
| Zeal-L/BiliBili-Manga-Downloader A GUI-based Bilibili Manga downloader built with Python and PySide6 that supports keyword search, QR-code login, multi-threaded downloads, … | 34 | 1248 | abandoned |
| syrusakbary/gdom GDOM is a Python CLI tool that lets you scrape and traverse web page DOM using GraphQL queries, built on the Graphene framework. You write … | 32 | 1239 | abandoned |
| chenjiandongx/mzitu A Python web crawler that downloads full photo galleries from mzitu.com, collecting over 3,000 image sets. It also performs word-frequency … | 32 | 1230 | abandoned |
| k1995/BaiduyunSpider A distributed crawler and search engine for Baidu Netdisk (Baidu Cloud) shared links, built with Scrapy and Scrapy-Redis, with a React-base… | 23 | 1182 | abandoned |
| xiaoguyu/wechatDownload A desktop application built with Electron, TypeScript, and Vue3 for downloading WeChat Official Account (公众号) articles. It captures require… | 10 | 1168 | abandoned |
| shadowmoose/RedditDownloader Reddit Media Downloader is a Python CLI tool that scans Reddit posts, comments, saved lists, and subreddits to download linked media locall… | 10 | 1156 | abandoned |
| xillwillx/skiptracer Skiptracer is a Python-based OSINT web scraping framework that aggregates paid and free lookup services to enumerate target information suc… | 23 | 1148 | abandoned |
| ring04h/weakfilescan A Python-based multi-threaded sensitive information leakage detection tool that crawls a target site, dynamically builds dictionary rules f… | 32 | 1140 | abandoned |
| joshhighet/ransomwatch A transparent ransomware claim tracker that monitors ransomware groups' leak sites on the dark web and records their victim claims. It aggr… | 10 | 1114 | abandoned |