Open-source crawlers software
569 open-source libraries, tools and applications tagged crawlers, ranked by how closely crawlers is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
569 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| NanmiCoder/MediaCrawler MediaCrawler is a Python-based multi-platform social media crawler that scrapes notes, videos, posts, and comments from Xiaohongshu, Douyin… | 72 | 65481 | active |
| soimort/you-get You-Get is a tiny command-line utility written in Python to download media content (videos, audios, images) from the web when no other hand… | 64 | 56873 | active |
| AIHawk AIHawk is an open-source Python AI agent that automates job applications by scraping job postings via browser automation and auto-applying … | 87 | 32150 | active |
| CloakBrowser CloakBrowser is a stealth Chromium distribution with 73 source-level C++ fingerprint patches that passes major bot-detection systems like C… | 82 | 32001 | active |
| iawia002/lux Lux is a fast and simple video downloader written in Go, available both as a CLI tool and a library. It supports downloading videos from ma… | 52 | 31775 | active |
| ScrapeGraphAI ScrapeGraphAI is a Python library that uses LLMs and direct graph logic to build web scraping pipelines for websites and local documents (X… | 87 | 31358 | active |
| ArchiveBox ArchiveBox is an open-source, self-hosted web archiving application that saves URLs, browser history, bookmarks, RSS feeds, and social medi… | 99 | 28737 | active |
| jhao104/proxy_pool A Python proxy pool service that periodically scrapes free proxies from 15+ sources, validates them, and stores them in Redis/SSDB. It expo… | 58 | 23762 | active |
| Builder.io A Node.js/TypeScript crawler that scrapes one or more websites and generates knowledge files (JSON) for creating custom GPTs or OpenAI assi… | 28 | 22405 | active |
| cat-catch Cat-catch is an open-source browser extension that sniffs and lists media resources (video, audio, images) loaded by the current web page. … | 90 | 22129 | active |
| Douyin_TikTok_Download_API A high-performance asynchronous Python service that scrapes and parses data from Douyin, TikTok, Kuaishou, and Bilibili, exposing a FastAPI… | 98 | 20610 | active |
| getmaxun/maxun Maxun is an open-source no-code platform for web scraping, crawling, search, and AI-powered data extraction that turns websites into struct… | 94 | 17844 | active |
| projectdiscovery/katana Katana is a fast, configurable web crawling and spidering framework written in Go, supporting both standard HTTP-based and headless browser… | 90 | 17565 | active |
| MODSetter/SurfSense SurfSense is an open-source, self-hostable NotebookLM alternative that combines a personal knowledge base with live open-web research conne… | 91 | 16505 | active |
| FlareSolverr FlareSolverr is a proxy server that bypasses Cloudflare and DDoS-GUARD protection by solving browser challenges with Selenium and undetecte… | 97 | 15906 | active |
| PuerkitoBio/goquery goquery is a Go library that provides a jQuery-like syntax for parsing, traversing, and manipulating HTML documents, built on Go's net/html… | 87 | 14996 | stable |
| pystardust/ani-cli ani-cli is a POSIX shell script CLI tool for browsing and watching anime from the terminal, scraping the anidb site and playing videos via … | 96 | 13851 | active |
| browserless/browserless Browserless is a platform for deploying and managing headless browsers (Chrome, Firefox, WebKit) in Docker, usable self-hosted or via their… | 98 | 13791 | active |
| instaloader/instaloader Instaloader is a Python command-line tool and library for downloading pictures, videos, captions, comments, and metadata from Instagram. It… | 91 | 13598 | active |
| JoeanAmier/XHS-Downloader A tool for extracting links and downloading content (images, videos, live photos) from XiaoHongShu (RedNote) posts. It ships as a Python TU… | 86 | 13039 | active |
| ultrafunkamsterdam/undetected-chromedriver A Python library that patches Selenium's ChromeDriver binary so automated Chrome sessions avoid detection by anti-bot systems like Cloudfla… | 42 | 12858 | active |
| daijro/camoufox Camoufox is an open-source anti-detect browser built on Firefox for web scraping and AI agents, with browser fingerprint spoofing and Playw… | 91 | 12641 | active |
| g1879/DrissionPage DrissionPage is a Python-based web automation library that combines browser control (via Chromium's DevTools Protocol, without WebDriver) w… | 70 | 12511 | active |
| Crawlab Crawlab is a Go-based distributed web crawler management platform with a web UI for managing, scheduling, and monitoring spiders written in… | 49 | 12278 | active |
| jina-ai/reader Jina AI Reader converts any URL into LLM-friendly markdown via the r.jina.ai prefix, and searches the web into markdown via s.jina.ai. It r… | 59 | 12158 | active |
| code4craft/webmagic WebMagic is a scalable web crawler framework for Java covering the full crawl lifecycle: downloading, URL management, content extraction (X… | 56 | 11670 | active |
| jo-inc/camofox-browser A self-hosted anti-detection browser server for AI agents, wrapping the Camoufox Firefox fork that spoofs fingerprints at the C++ level. It… | 83 | 11663 | active |
| jhy/jsoup jsoup is a Java library for parsing, manipulating, and cleaning real-world HTML and XML, implementing the WHATWG HTML5 specification to pro… | 93 | 11410 | stable |
| ihmily/DouyinLiveRecorder A Python-based livestream recording application that can run unattended in a loop and record multiple streams simultaneously from 40+ platf… | 59 | 11051 | active |
| Thysrael/Horizon Horizon is an AI-powered news aggregation platform that fetches content from sources like Hacker News, GitHub, RSS, Reddit, and Telegram, s… | 61 | 10909 | active |
| pinchtab/pinchtab PinchTab is a standalone Go HTTP server that gives AI agents direct control over Chrome via CDP, with stealth injection and multi-instance … | 79 | 10299 | active |
| Apify Agent Skills The Apify MCP Server exposes thousands of Apify Store scrapers, crawlers, and automation tools to AI agents via the Model Context Protocol,… | 83 | 9896 | active |
| dataabc/weiboSpider A Python command-line crawler that scrapes posts and profile data from Sina Weibo users, writing results to txt/csv/json files or MySQL/Mon… | 58 | 9733 | active |
| kepano/defuddle Defuddle is a TypeScript library, CLI, and hosted service that extracts the main content from web pages, removing clutter like comments, si… | 88 | 9723 | active |
| qiye45/wechatDownload A desktop tool for batch downloading WeChat Official Account (公众号) articles, saving them as html/mhtml/md/pdf/docx/csv and preserving embed… | 84 | 9560 | active |
| RSS-Bridge/rss-bridge RSS-Bridge is a PHP web application that generates RSS, Atom, and JSON feeds for websites that don't offer one, using hundreds of site-spec… | 75 | 9295 | active |
| eze-is/web-access An Agent Skill that gives AI coding agents (Claude Code, Cursor, Gemini CLI, etc.) full web access capabilities: three-tier channel dispatc… | 63 | 9124 | active |
| HerbertHe/iptv-sources A service that automatically aggregates and updates IPTV m3u playlist sources from multiple public repositories, with EPG data and optional… | 72 | 9026 | active |
| hardikvasa/google-images-download A Python command-line tool that searches and downloads hundreds of images from Google Images to local storage. It uses Selenium with Chrome… | 66 | 8695 | active |
| kangvcar/InfoSpider InfoSpider is an open-source Python toolbox that crawls a user's own personal data from dozens of Chinese and international services (email… | 54 | 8264 | active |
| cv-cat/Spider_XHS A Python library that reverse-engineers Xiaohongshu (Little Red Book) signature algorithms and wraps the platform's PC, creator, and Pugong… | 89 | 8053 | active |
| alirezamika/autoscraper AutoScraper is a Python library that automatically learns scraping rules from a URL or HTML content plus a list of sample data you want to … | 62 | 8003 | stable |
| Steel Browser Steel Browser is an open-source browser API and sandbox that manages browser sessions, proxies, stealth, and lifecycle so developers can bu… | 86 | 7708 | active |
| hect0x7/JMComic-Crawler-Python A Python library providing an API client for the JMComic (18comic) site, supporting both web and mobile endpoints, with album downloading, … | 98 | 7607 | stable |
| eRPC Pholcus is a distributed, high-concurrency web crawler framework written in pure Go. It supports standalone, server, and client modes with … | 84 | 7582 | active |
| mgdm/htmlq htmlq is a command-line tool, like jq but for HTML, that extracts content from HTML documents using CSS selectors. Written in Rust, it read… | 57 | 7580 | stable |
| berstend/puppeteer-extra A modular plugin framework for Puppeteer (and Playwright via playwright-extra) that extends headless browser automation with drop-in plugin… | 32 | 7406 | active |
| autoscrape-labs/pydoll Pydoll is a Python library for automating Chromium-based browsers directly over the Chrome DevTools Protocol, with no WebDriver binary and … | 84 | 7107 | active |
| mishushakov/llm-scraper A TypeScript library that turns any webpage into structured data using LLMs, built on Playwright and the Vercel AI SDK. It supports multipl… | 68 | 6937 | active |
| adbar/trafilatura Trafilatura is a Python package and command-line tool for crawling the web and extracting main text, metadata, and comments from raw HTML w… | 94 | 6914 | active |
| bda-research/node-crawler node-crawler is a TypeScript web crawler/spider library for Node.js that fetches pages and provides server-side DOM parsing with automatic … | 84 | 6794 | active |
| VeNoMouS/cloudscraper A Python library that wraps Requests to bypass Cloudflare's anti-bot protection pages (IUAM), supporting challenge types v1, v2, v3, and Tu… | 30 | 6776 | active |
| curl-impersonate bindings curl_cffi is a Python binding for a curl-impersonate fork via cffi, providing an HTTP client that can impersonate browser TLS/JA3, HTTP/2, … | 99 | 6746 | active |
| davidteather/TikTok-Api An unofficial Python API wrapper for TikTok.com that retrieves trending content, user information, hashtags, and video data without authent… | 85 | 6681 | active |
| hect0x7/JMComic-APK A GitHub Actions-based automation that periodically checks for updates to the JM Comic (18comic) Android APK and publishes new versions as … | 93 | 6589 | active |
| gosom/google-maps-scraper An open-source Go tool that scrapes Google Maps to extract business data such as names, addresses, phone numbers, websites, ratings, review… | 95 | 6453 | active |
| chyroc/WechatSogou A Python library providing a scraping API for WeChat official accounts based on Sogou WeChat search. It lets you look up account info and r… | 51 | 6398 | active |
| Bright Data MCP The official Bright Data CLI (npm package @brightdata/cli) providing terminal access to Bright Data's web scraping, search, and structured … | 81 | 6261 | active |
| Python3WebSpider/ProxyPool A self-hosted proxy pool service that periodically scrapes free proxy sites, stores and scores them in Redis, tests their availability, and… | 60 | 6252 | active |
| bb-browser bb-browser is a TypeScript CLI and MCP server that lets AI agents control your real Chrome browser using your existing login state, exposin… | 64 | 6241 | active |
| BrowserAct BrowserAct Skills is a Python-based browser automation CLI designed for AI agents, providing real-browser control with anti-bot evasion (st… | 57 | 6118 | active |
| MontFerret/ferret Ferret is a declarative, expression-oriented query language (FQL) with an embeddable Go runtime for querying, transforming, and automating … | 79 | 6013 | active |
| AhmadIbrahiim/Website-downloader A Node.js web application that downloads the complete source code of any website, including all assets like JavaScripts, stylesheets, and i… | 72 | 5980 | active |
| omkarcloud/botasaurus Botasaurus is an all-in-one Python web scraping framework with built-in anti-detection, caching, parallelization, and proxy support. It let… | 69 | 5721 | active |
| Pixiv Batch Downloader A browser extension (Chrome, Edge, Firefox) for batch downloading illustrations, manga, Ugoira animations, and novels from Pixiv. It offers… | 97 | 5664 | active |
| rmax/scrapy-redis Redis-based components for Scrapy that enable distributed crawling and scraping by sharing a Redis queue across multiple spider instances. … | 66 | 5644 | active |
| Yuukiy/JavSP JavSP is a Python command-line tool that scrapes adult video (JAV) metadata from multiple websites, aggregates the data, and generates NFO … | 23 | 5173 | active |
| lc/gau gau (getallurls) is a Go CLI tool that fetches known URLs for a given domain from AlienVault's Open Threat Exchange, the Wayback Machine, C… | 67 | 5127 | active |
| DedSecInside/TorBot TorBot is a Python CLI tool for OSINT on the dark web, crawling .onion sites over the Tor network and building link trees. It can save craw… | 95 | 5045 | active |
| jaypyles/Scraperr Scraperr is a self-hosted web scraping application with a web UI that lets users scrape websites without writing code, using XPath-based ex… | 10 | 4910 | active |
| Patchright Patchright is a patched, undetected fork of the Playwright browser automation framework that evades bot-detection systems like Cloudflare. … | 95 | 4904 | active |
| MechanicalSoup/MechanicalSoup A Python library for automating interaction with websites, built on Requests and BeautifulSoup. It handles cookies, redirects, link followi… | 62 | 4894 | active |
| rachelos/we-mp-rss A self-hosted WeChat official account (公众号) subscription assistant that scrapes articles, generates RSS feeds, and converts content to Mark… | 83 | 4868 | active |
| ultrafunkamsterdam/nodriver Nodriver is a fully asynchronous Python browser automation and web scraping library, and the official successor to Undetected-Chromedriver.… | 59 | 4799 | active |
| Keiyoushi Extensions A community-maintained repository of extensions (APKs) for Mihon and its forks, providing manga source plugins. The source code for the ext… | 72 | 4797 | active |
| HTTrack Website Copier HTTrack is a free offline browser utility that recursively downloads websites to a local directory, rewriting links so the mirrored copy ca… | 99 | 4780 | stable |
| l0o0/translators_CN A community-maintained collection of Zotero translators for Chinese academic and general websites, enabling Zotero to scrape citation metad… | 76 | 4776 | active |
| bjesus/pipet Pipet is a command-line web scraper written in Go that extracts data from online assets using HTML parsing, JSON parsing, and client-side J… | 14 | 4768 | active |
| Twikit Twikit is a free Python library that wraps Twitter's internal API, allowing posting, searching, and scraping tweets without an official API… | 57 | 4720 | active |
| lecepin/WeChatVideoDownloader A convenient desktop GUI application for downloading videos from WeChat Channels (WeChat Video Accounts). It intercepts and captures video … | 10 | 4669 | active |
| dataabc/weibo-crawler A Python crawler for Sina Weibo that scrapes user profiles and posts, exporting data to CSV, JSON, MySQL, MongoDB, or SQLite, and optionall… | 70 | 4663 | active |
| joeyism/linkedin_scraper A Python library that scrapes LinkedIn for user, company, and job data using Playwright with an async API. It provides Pydantic data models… | 75 | 4603 | active |
| Anakin-Inc/anakin AnakinScraper OSS is a self-hosted web scraping API written in Go that turns any website into LLM-ready markdown or structured JSON via a s… | 68 | 4535 | active |
| speedyapply/JobSpy JobSpy is a Python library that scrapes job postings from popular job boards like LinkedIn, Indeed, Glassdoor, Google, and ZipRecruiter con… | 97 | 4514 | active |
| sparklemotion/mechanize Mechanize is a Ruby library for automating interaction with websites. It handles cookies, redirects, link following, and form submission wh… | 82 | 4443 | stable |
| Lucksi/Mr.Holmes Mr.Holmes is a Python-based OSINT (open-source intelligence) CLI tool that gathers information about usernames, domains, phone numbers, and… | 50 | 4314 | active |
| kanasimi/work_crawler A multi-language downloader application that batch-downloads web novels (converting them to EPUB) and comics from a large list of Chinese, … | 67 | 4303 | active |
| UltimaHoarder/UltimaScraper A Python-based scraper that downloads all media (photos, videos) from OnlyFans accounts using the user's own session authentication. It sto… | 23 | 4278 | active |
| dotnetcore/DotnetSpider DotnetSpider is a .NET Standard web crawling and scraping framework that is lightweight, efficient, and cross-platform. It supports distrib… | 53 | 4144 | active |
| RipMeApp/ripme RipMe is a cross-platform Java application that bulk-downloads image albums from websites like Reddit, Imgur, Twitter, Instagram, and Tumbl… | 92 | 4126 | active |
| nghuyong/WeiboSpider A continuously maintained Python web scraping tool for Sina Weibo built on Scrapy and the new weibo.com API. It collects user profiles, pos… | 69 | 4119 | active |
| IonicaBizau/scrape-it scrape-it is a Node.js web scraping library with a simple, declarative API for extracting data from HTML pages, built on top of tinyreq and… | 99 | 4069 | active |
| fake-useragent/fake-useragent A Python library that generates realistic, up-to-date browser user-agent strings from a bundled real-world database. It supports random or … | 10 | 4049 | active |
| MikeChongCan/scylla Scylla is an intelligent proxy pool service that automatically crawls, validates, and serves proxy IPs via a JSON API and web UI. It integr… | 30 | 4019 | active |
| JoeanAmier/TikTokDownloader DouK-Downloader (formerly TikTokDownloader) is an open-source Python tool for downloading and scraping content from Douyin and TikTok, incl… | 88 | 16930 | maintenance |
| binux/pyspider pyspider is a powerful web crawler (spider) system written in Python with a built-in WebUI for script editing, task monitoring, project man… | 10 | 16755 | maintenance |
| kingname/GeneralNewsExtractor GNE (GeneralNewsExtractor) is a Python library that extracts article title, author, publish time, body text, and images from news page HTML… | 75 | 3802 | active |
| edoardottt/cariddi Cariddi is a fast command-line web crawler written in Go that takes a list of domains, crawls URLs, and scans for endpoints, secrets, API k… | 84 | 3786 | active |
| feapder feapder is a powerful, easy-to-use Python web crawling framework offering four spider types (AirSpider, Spider, TaskSpider, BatchSpider) fo… | 68 | 3736 | active |
| oxylabs/google-ai-mode-scraper A code repository of examples for Oxylabs' Google AI Mode Scraper, a commercial API that sends prompts to Google AI Mode and returns parsed… | 59 | 3626 | active |
page 1 / 6 next →