Open-source web-scraping software
2000 open-source libraries, tools and applications tagged web-scraping, ranked by how closely web-scraping is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
2000 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| zhangyd-c/OneBlog OneBlog is a self-hosted Java blog system built with Spring Boot and Bootstrap, featuring a full admin backend and responsive public-facing… | 24 | 1105 | active |
| bountyyfi/lonkero Lonkero is a professional-grade web application security scanner written in Rust, built for real penetration testing with 125+ scan modules… | 67 | 1103 | active |
| Vigolium Vigolium is a high-fidelity web vulnerability scanner written in Go that combines deterministic multi-phase scanning (317 modules for conte… | 80 | 1099 | active |
| pa11y/pa11y-dashboard Pa11y Dashboard is a self-hosted Node.js web interface for the Pa11y accessibility reporter that automatically tests web pages for accessib… | 73 | 1090 | active |
| y-scope/clp CLP (Compressed Log Processor) is an open-source log management tool that losslessly compresses JSON and unstructured text logs and enables… | 94 | 1088 | active |
| XInTheDark/raycast-g4f A Raycast extension that provides free access to GPT-4, Claude, Llama, Gemini and other AI models via GPT4Free providers, with full support… | 51 | 1087 | active |
| christian-fei/my-yt A self-hosted, minimal YouTube frontend built with vanilla JavaScript and yt-dlp that lets you manage channel subscriptions, download video… | 65 | 1074 | active |
| trendy-design/llmchat LLMChat is a privacy-focused AI chat platform built with Next.js and TypeScript that provides a unified interface for chatting with multipl… | 21 | 1074 | active |
| klsdf/GreenResourcesManager Green Resources Manager is a free, all-in-one local multimedia resource manager for 'data hoarder' users, built with Electron and Vue 3. It… | 84 | 1058 | active |
| apify/proxy-chain A programmable HTTP/HTTPS proxy server library for Node.js, similar to Squid, with support for SSL/TLS, SOCKS4/5, authentication, upstream … | 93 | 1044 | active |
| KARMA karma_v2 is a Bash-based passive OSINT reconnaissance framework that automates Shodan queries to enumerate assets, exposed services, CVEs, … | 39 | 1030 | active |
| BGmi/BGmi BGmi is a Python CLI tool with a Web UI for subscribing to and automatically downloading bangumi (anime) episodes via torrent sources like … | 89 | 1026 | active |
| NoOne-hub/JSReverser-MCP JSReverser-MCP is an MCP server that standardizes JavaScript reverse-engineering workflows by exposing browser observation, script search, … | 66 | 1017 | active |
| binux/qiandao qiandao is an automated check-in (sign-in) framework based on a HAR editor, letting users record HTTP requests and replay them on a schedul… | 32 | 3429 | maintenance |
| LLM-As-Chatbot A Python application that serves open-source instruction-tuned LLMs as chatbot services via a Gradio web UI or a Discord bot. It integrates… | 31 | 3315 | maintenance |
| mnutt/hummingbird Hummingbird is a self-hosted real-time web analytics application built on Node.js that tracks visitor activity via a 1x1 tracking pixel and… | 57 | 2932 | maintenance |
| SCUTlihaoyu/open-chat-video-editor An open-source Python tool that automatically generates short videos from a short text prompt or a web URL, producing narration, background… | 30 | 2816 | maintenance |
| m0rtem/CloudFail CloudFail is a Python 3 command-line reconnaissance tool that attempts to discover the real IP address of servers hidden behind Cloudflare.… | 32 | 2698 | maintenance |
| shirosaidev/stocksight stocksight is an open-source Python application that analyzes stock market sentiment using Twitter streams and news headlines, storing data… | 23 | 2541 | maintenance |
| filipedeschamps/video-maker An open-source Node.js project that automates video production by chaining robots that fetch content, analyze text with IBM Watson NLU, and… | 32 | 2493 | maintenance |
| alephdata/aleph Aleph is a self-hosted platform for indexing, searching, and browsing large volumes of documents (PDF, Word, HTML) and structured data (CSV… | 63 | 2441 | maintenance |
| retroplasma/earth-reverse-engineering A reverse-engineering project documenting and implementing access to Google Earth's undocumented 3D satellite mode, including URL structure… | 10 | 2321 | maintenance |
| thehackingsage/hacktronian Hacktronian is a Python-based, menu-driven collection of penetration testing tools that bundles popular utilities for information gathering… | 32 | 2204 | maintenance |
| LionSec/xerosploit Xerosploit is a Ruby-based penetration testing toolkit that wraps bettercap and nmap to perform man-in-the-middle attacks, port scanning, a… | 23 | 2204 | maintenance |
| FeeiCN/GSIL GSIL is a Python tool that monitors GitHub for sensitive information leaks, such as leaked credentials, internal domains, and proprietary c… | 10 | 2146 | maintenance |
| s0md3v/ReconDog ReconDog is a Python-based reconnaissance 'Swiss Army Knife' that gathers information about targets (domains, IPs) using third-party APIs l… | 23 | 2115 | maintenance |
| rendora/rendora Rendora is a dynamic renderer written in Go that acts as a reverse HTTP proxy, serving headless-Chrome-rendered HTML to web crawlers while … | 10 | 1987 | maintenance |
| circlestarzero/EX-chatGPT Ex-ChatGPT is a Python web application that lets ChatGPT call external APIs (Google, WolframAlpha, WikiMedia) to give more accurate, up-to-… | 10 | 1956 | maintenance |
| w-digital-scanner/w13scan W13Scan is an open-source Python3 web vulnerability scanner supporting both passive (proxy-based) and active scanning modes. It ships with … | 32 | 1944 | maintenance |
| nat/natbot natbot is a Python tool that drives a web browser using GPT-3, letting an LLM agent navigate and interact with web pages. It serializes the… | 32 | 1933 | maintenance |
| jaykali/hackerpro HackerPro is an all-in-one penetration testing tool collection for Linux and Android (Termux) that bundles popular security tools like Nmap… | 32 | 1879 | maintenance |
| 0xInfection/TIDoS-Framework TIDoS is a Python-based offensive web application penetration testing framework with a Metasploit-like console interface and an optional Qt… | 23 | 1869 | maintenance |
| iceyhexman/onlinetools A self-hosted web-based penetration testing toolbox written in Python that bundles common recon and scanning tasks behind a browser UI. It … | 10 | 1780 | maintenance |
| nashsu/FreeAskInternet FreeAskInternet is a self-hosted, Perplexity-like search aggregator and answer generator that runs multi-engine searches via SearXNG and fe… | 25 | 8749 | experimental |
| al0ne/Vxscan Vxscan is a Python3-based comprehensive security scanning tool for authorized penetration testing. It combines host liveness checks, port s… | 32 | 1758 | maintenance |
| CyberPunkMetalHead/Binance-News-Sentiment-Bot A Python trading bot that analyzes news headlines from the top 100 crypto news feeds and places Binance buy/sell orders based on sentiment … | 32 | 1652 | maintenance |
| altryne/chatGPT-telegram-bot A Telegram bot that lets users chat with ChatGPT, originally by driving the ChatGPT web interface in Chromium via Playwright, with plans to… | 23 | 1637 | maintenance |
| sairson/Yasso Yasso is a Go-based intranet penetration testing toolkit that combines service brute-forcing (RDP, SSH, Redis, PostgreSQL, MongoDB, MSSQL, … | 23 | 1595 | maintenance |
| Frrrrrrrrank/auto_job__find__chatgpt__rpa A free Python RPA script that uses OpenAI's API to automatically generate cover letters from your resume and job descriptions, then sends m… | 27 | 1571 | maintenance |
| SharadKumar97/OSINT-SPY OSINT-SPY is a Python command-line tool that performs open-source intelligence scans on emails, domains, IP addresses, organizations, Bitco… | 23 | 1550 | maintenance |
| lukeaschenbrenner/TxtNet-Browser TxtNet Browser is an Android app that lets users browse the web over SMS without any WiFi or mobile data connection. It sends HTTP GET requ… | 75 | 1541 | maintenance |
| gwen001/github-search A collection of Python, PHP, and Bash scripts that perform targeted searches on GitHub via its search API to find secrets, keys, private re… | 23 | 1515 | maintenance |
| run-llama/llama-lab Llama Lab is a collection of experimental projects built on top of LlamaIndex for exploring cutting-edge LLM agent capabilities. It include… | 30 | 1508 | maintenance |
| SamJoan/droopescan Droopescan is a plugin-based command-line scanner that helps security researchers identify the CMS, version, plugins, themes, and interesti… | 32 | 1450 | maintenance |
| grimd Grimd is a fast DNS proxy server written in Go that blackholes advertisement and malware domains using configurable blocklists. It runs any… | 23 | 1415 | maintenance |
| TideSec/Mars Mars is a self-hosted security platform for asset discovery, subdomain enumeration, port and web fingerprinting, and change monitoring of i… | 32 | 1383 | maintenance |
| lijiejie/swagger-exp A Python-based Swagger REST API information disclosure exploitation tool. It enumerates API endpoints, auto-fills parameters, tests for una… | 32 | 1379 | maintenance |
| chizhu/KGQA_HLM A knowledge-graph-based question answering and visualization system for character relationships in the Chinese classic novel 'Dream of the … | 32 | 1339 | maintenance |
| yangxuanxc/wechat_friends A Python application that analyzes WeChat friend data (gender, location, signatures, nicknames, avatars) and visualizes it with charts, map… | 32 | 1327 | maintenance |
| infobyte/evilgrade Evilgrade is a modular penetration testing framework that exploits poor software update implementations by injecting fake updates via DNS m… | 23 | 1326 | maintenance |
| k8gege/K8CScan K8CScan is a high-concurrency, plugin-based scanner designed for large internal network penetration testing. It bundles information gatheri… | 32 | 1310 | maintenance |
| hangetzzu/saycheese SayCheese is a shell-based social engineering tool that generates a malicious HTTPS page to capture webcam photos from a target who clicks … | 32 | 1285 | maintenance |
| dragnet-org/dragnet Dragnet is a Python library that uses machine learning models to extract the main article content, and optionally user comments, from HTML … | 33 | 1277 | maintenance |
| AbirHasan2005/ShellPhish A modified (modded) version of the ShellPhish phishing simulation tool that generates fake login pages for popular websites like Facebook, … | 32 | 1272 | maintenance |
| W01fh4cker/Serein Serein is a graphical Python tool for batch-collecting URLs via the FOFA search engine API and running batch detection/exploitation of know… | 10 | 1248 | maintenance |
| m4n3dw0lf/pythem pythem is a multi-purpose penetration testing framework written in Python 2.7, providing an interactive CLI for security researchers. It bu… | 32 | 1247 | maintenance |
| dionach/CMSmap CMSmap is a Python open-source CLI scanner that automates detection of security flaws in popular CMSs, integrating common vulnerabilities f… | 32 | 1185 | maintenance |
| SpiderLabs/HostHunter HostHunter is a Python CLI recon tool that maps IPv4/IPv6 targets to virtual hostnames using OSINT and active reconnaissance techniques suc… | 23 | 1172 | maintenance |
| d3ckx1/Fvuln Fvuln (Find-Vulnerability) is an automated security scanning tool for penetration testers and red teams. It combines live IP detection, por… | 23 | 1162 | maintenance |
| hausec/ADAPE-Script A PowerShell script that automates Active Directory assessment and privilege escalation checks by bundling multiple well-known pentest modu… | 32 | 1127 | maintenance |
| mdsecactivebreach/o365-attack-toolkit A Go-based red team toolkit for performing OAuth phishing attacks against Office365 accounts. It uses stolen tokens with the Microsoft Grap… | 32 | 1124 | maintenance |
| remy/inliner A Node.js utility and CLI that converts a web page into a single self-contained HTML file by inlining CSS, JavaScript, images, and videos (… | 23 | 1117 | maintenance |
| DasSecurity-HatLab/AoiAWD AoiAWD is a lightweight, portable EDR (Endpoint Detection and Response) system designed for CTF Attack-Defense (AWD) competitions. It captu… | 32 | 1111 | maintenance |
| Syan-Lin/CyberWaifu CyberWaifu is a Python chatbot application that combines LLMs (ChatGPT, Claude) with TTS (edge-tts, Azure) to create realistic conversation… | 30 | 1102 | maintenance |
| facebookresearch/cc_net CCNet is a Python pipeline from Facebook AI Research for downloading, deduplicating, and cleaning Common Crawl web data into high-quality m… | 10 | 1045 | maintenance |
| ledccn/IYUUPlus IYUUPlus is a self-hosted auto-reseeding tool for Private Tracker (PT) torrents, running as a resident service with a WebUI. It matches exi… | 10 | 1039 | maintenance |
| hzlzh/Alfred-Workflows A collection of Alfred 2 workflows, extensions, and themes for macOS, including URL shortening, SEO checking, file path copying, and workfl… | 23 | 1033 | maintenance |
| zabirauf/AutoGPT.js AutoGPT.js is an open-source web application that runs AutoGPT-style autonomous GPT agents directly in the browser. It supports file creati… | 31 | 1024 | maintenance |
| springboot-dubbox A Java distributed backend system built on Spring Boot and DubboX, providing REST APIs and an admin management system with role-based acces… | 32 | 1015 | maintenance |
| Significant-Gravitas/Auto-GPT-Plugins A collection of official and community plugins for Auto-GPT, the autonomous AI agent framework, providing additional capabilities like web … | 10 | 3821 | experimental |
| pingcap/autoflow AutoFlow is an open-source Graph RAG knowledge base tool that builds knowledge graphs over crawled documentation and answers questions via … | 60 | 2974 | experimental |
| semanser/codel Codel is a self-hosted, fully autonomous AI agent that executes complex tasks using a sandboxed Docker environment with terminal, browser, … | 16 | 2476 | experimental |
| idflood/ThreeNodes.js ThreeNodes.js is a browser-based node-based visual programming environment inspired by vvvv, built with JavaScript, HTML, and WebGL (three.… | 32 | 2346 | experimental |
| s0md3v/Striker Striker is a Python-based offensive reconnaissance and vulnerability scanning suite that discovers subdomains, scans common ports, detects … | 23 | 2344 | experimental |
| kenn-io/msgvault msgvault is a Go application that archives a lifetime of email, chat, calendar, and meeting data into a local SQLite/DuckDB store, supporti… | 83 | 2160 | experimental |
| Cormanz/smartgpt SmartGPT is an experimental Rust program that lets LLMs like GPT-3.5 and GPT-4 autonomously complete complex tasks by breaking them into su… | 31 | 1737 | experimental |
| obsei/obsei Obsei is an open-source, low-code, AI-powered automation framework for text analysis workflows. It collects unstructured data from sources … | 99 | 1437 | experimental |
| cinderline/northcinder NorthCinder is a self-hosted MCP server that acts as a buyer-side shopping agent, comparing products across user-chosen sources and explain… | 73 | 1208 | experimental |
| Omniplex-ai/omniplex Omniplex is an open-source, self-hosted alternative to Perplexity, an AI-powered answer engine that combines web search with LLMs to genera… | 25 | 1060 | experimental |
| zedeus/nitter Nitter is a free and open-source alternative Twitter/X front-end focused on privacy and performance, written in Nim. It proxies all request… | 10 | 14486 | abandoned |
| GoogleChrome/rendertron Rendertron is a standalone HTTP service that renders and serializes web pages on the fly using headless Chrome via Puppeteer, so bots that … | 10 | 5950 | abandoned |
| Libreddit Libreddit is a private, lightweight front-end for Reddit written in Rust that proxies all requests (including media) through the server so … | 23 | 5198 | abandoned |
| qq547276542/Agriculture_KnowledgeGraph AgriKG is an agricultural domain knowledge graph system built on Neo4j with a Django web interface, including crawled encyclopedia data, na… | 32 | 4395 | abandoned |
| Arachni Arachni is a modular, high-performance Ruby framework for scanning web applications for security vulnerabilities, including XSS and SQL inj… | 10 | 4041 | abandoned |
| livid/v2ex-gae Project Babel 2 is the legacy Python codebase that powered the V2EX community, a forum application for creating and supporting online discu… | 10 | 3082 | abandoned |
| legba XRay is a Go-based CLI tool for network reconnaissance and OSINT that enumerates subdomains via DNS bruteforcing, gathers open-port intelli… | 10 | 2331 | abandoned |
| RD17/ambar Ambar is an open-source, self-hosted document search engine with automated file system crawling, OCR, tagging, and instant Google-like full… | 10 | 1943 | abandoned |
| detro/ghostdriver GhostDriver is a pure JavaScript implementation of the WebDriver Wire Protocol that uses PhantomJS as a headless browser back-end, exposing… | 10 | 1908 | abandoned |
| n4ze3m/dialoqbase Dialoqbase is an open-source self-hosted application for creating custom chatbots powered by a personal knowledge base, supporting many cha… | 10 | 1791 | abandoned |
| xdavidhu/mitmAP A Python program that creates a fake wireless access point and performs man-in-the-middle data sniffing using tools like SSLstrip2, mitmpro… | 10 | 1692 | abandoned |
| insoxin/imaotai A Dockerized automation service that performs daily automatic reservations for the i茅台 (iMaotai) app, scheduling purchases of Moutai liquor… | 10 | 1545 | abandoned |
| fossasia/open-event-scraper A Python tool that parses Google spreadsheets and converts them into Open Event JSON format, originally built for FOSSASIA 2016 event data.… | 10 | 1516 | abandoned |
| KnowledgeCanvas/knowledge Knowledge is a desktop application for saving, searching, exploring, and chatting with websites, documents, and files using a knowledge gra… | 76 | 1462 | abandoned |
| Medium/phantomjs An NPM package (phantomjs-prebuilt) that downloads and installs a blessed version of the PhantomJS headless WebKit browser and exposes its … | 10 | 1430 | abandoned |
| paicha/gxgk-wechat-server A campus WeChat official account backend built with Python, Flask, Redis, MySQL, and Celery, providing students with services like grade lo… | 10 | 1381 | abandoned |
| mathdroid/covid-19-api A JSON API service serving global COVID-19 data sourced from Johns Hopkins University CSSE, built with TypeScript for the ZEIT Now (Vercel)… | 23 | 1349 | abandoned |
| dmotz/thing-translator Thing Translator is a web app that lets users point their phone or laptop camera at objects to hear how to say their names in a different l… | 32 | 1328 | abandoned |
| Kamerka Kamerka is a Python CLI script that builds an interactive map of internet-exposed cameras, printers, tweets, and photos around given coordi… | 10 | 1264 | abandoned |
| putianxi/putianxi.github.io A Vue.js-based static website that maps and lists Putian-system hospitals across China to raise public awareness of their risks. It renders… | 23 | 1235 | abandoned |
| SchizoDuckie/DuckieTV DuckieTV is a standalone desktop application for TV show addicts that provides a personalized TV-show calendar with integrated, blocking-re… | 83 | 1190 | abandoned |
← prev page 20 / 20