Open-source web-scraping software
2000 open-source libraries, tools and applications tagged web-scraping, ranked by how closely web-scraping is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
2000 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| spacedriveapp/spacedrive Spacedrive is an open-source cross-platform file explorer powered by a virtual distributed filesystem written in Rust. It indexes files acr… | 67 | 39079 | active |
| khoj-ai/khoj Khoj is a self-hostable personal AI assistant ('second brain') that answers questions from your own documents and the web using local or on… | 83 | 37738 | active |
| ItzCrazyKns/Vane Vane is a self-hosted, privacy-focused AI answering engine that combines web search (via SearxNG) with LLMs to deliver cited answers. It su… | 79 | 37362 | active |
| reworkd/AgentGPT AgentGPT is a web application for assembling, configuring, and deploying autonomous AI agents directly in the browser. Users name a custom … | 10 | 36296 | active |
| CyberChef CyberChef is a browser-based web application for performing a wide range of 'cyber' operations such as encryption, encoding, compression, h… | 98 | 36120 | active |
| stanford-oval/storm STORM is an LLM-powered system that researches a topic via internet search and generates full-length, cited, Wikipedia-like reports. It is … | 32 | 31520 | active |
| GPT Researcher GPT Researcher is a Python‑based autonomous AI agent that performs deep, multi‑source research and generates detailed reports with citation… | 95 | 29943 | active |
| Fosowl/agenticSeek AgenticSeek is a fully local, privacy-focused AI assistant that serves as an open-source alternative to Manus AI. It runs reasoning models … | 64 | 27482 | active |
| every-app/open-seo OpenSEO is an open‑source SEO platform that provides keyword research, rank tracking, backlink analysis, site audits and AI visibility tool… | 82 | 23344 | active |
| 1Panel-dev/MaxKB MaxKB is an open-source, self-hosted platform for building enterprise-grade AI agents with RAG pipelines, workflow orchestration, and MCP t… | 96 | 22992 | active |
| FreeTube FreeTube is an open-source desktop YouTube client built with privacy in mind, letting users watch videos without ads or Google tracking. Al… | 86 | 22073 | active |
| MobSF/Mobile-Security-Framework-MobSF MobSF is an automated all-in-one mobile application security testing framework for Android, iOS, and Windows Mobile apps. It performs stati… | 94 | 21980 | active |
| Alibaba-NLP/DeepResearch Tongyi DeepResearch is an open-source agentic large language model (30.5B total parameters, 3.3B activated per token) plus training and inf… | 49 | 20014 | active |
| WLED WLED is a fast, feature-rich open-source firmware for ESP32 (and legacy ESP8266) microcontrollers that controls addressable RGB LED strips … | 89 | 18773 | active |
| Jackett/Jackett Jackett is a self-hosted proxy server that translates queries from media automation apps (Sonarr, Radarr, Lidarr, etc.) into tracker-site-s… | 95 | 16216 | active |
| eigent-ai/eigent Eigent is an open-source desktop application for building and running AI agent workforces that automate complex multi-step workflows. Built… | 85 | 15506 | active |
| yusufkaraaslan/Skill_Seekers Skill Seekers is a Python CLI toolkit and MCP server that converts documentation sites, GitHub repos, PDFs, codebases, notebooks, and other… | 83 | 15096 | active |
| PanSou PanSou is a high-performance self-hosted API service for searching cloud drive (netdisk) resource links, aggregating results from Telegram … | 63 | 14913 | active |
| Audiobookshelf Audiobookshelf is an open-source, self-hosted media server for streaming and managing audiobooks and podcasts, with companion Android and i… | 98 | 14554 | active |
| myhhub/stock InStock is a Python-based stock analysis system for Chinese A-shares and ETFs that captures daily market data, computes technical indicator… | 61 | 14527 | active |
| Playwright A command-line interface for Playwright browser automation designed for coding agents like Claude Code and GitHub Copilot. It provides toke… | 99 | 14472 | active |
| nanobrowser/nanobrowser Nanobrowser is an open-source Chrome extension that provides AI-powered web automation through a multi-agent system (Planner, Navigator, et… | 67 | 13818 | active |
| Gotenberg Gotenberg is a Docker-based HTTP API for converting documents (HTML, URLs, Markdown, Office files) into PDFs using headless Chromium and Li… | 97 | 13288 | stable |
| Stash Stash is a self-hosted web application written in Go that organizes, tags, and streams personal video and image collections through a brows… | 88 | 13034 | active |
| Open Deep Research An open-source deep research agent built with LangGraph that performs multi-step web research using configurable LLM providers, search tool… | 10 | 12680 | active |
| calesthio/Crucix Crucix is a self-hosted OSINT intelligence terminal that aggregates 27 open-source data feeds (satellite fire detection, flight tracking, r… | 49 | 12090 | active |
| Scira Scira is an open-source, minimalistic AI-powered search engine and agentic research platform that plans multi-step searches, retrieves live… | 64 | 11904 | active |
| chaitin/xray xray is a security assessment tool from Chaitin that scans web applications for common vulnerabilities like XSS and SQL injection, supporti… | 23 | 11752 | active |
| handsomestWei/patent-disclosure-skill A Python-based agent skill for Chinese patent work that mines patentable points from project materials, drafts disclosure documents (invent… | 60 | 11694 | active |
| UI-TARS UI-TARS is ByteDance's open-source multimodal AI agent stack, comprising Agent TARS (a CLI/Web UI multimodal agent that operates terminals,… | 46 | 11567 | active |
| perspective-dev/perspective Perspective is an interactive analytics and data visualization component for large, real-time, and streaming datasets, built around a high-… | 99 | 11296 | active |
| BrasilAPI/BrasilAPI BrasilAPI is a free public REST API that centralizes Brazilian public data (CEP postal codes, and other government-adjacent data) behind mo… | 76 | 11259 | active |
| linkding linkding is a self-hosted bookmark manager built with Django, designed to be minimal, fast, and easy to deploy via a single Docker containe… | 96 | 11255 | stable |
| simplifaisoul/osiris OSIRIS is an open-source, real-time global intelligence (OSINT) dashboard that aggregates live flight tracking, maritime, CCTV, seismic, fi… | 70 | 11174 | active |
| NopeCHA NopeCHA is an AI-powered CAPTCHA solving service distributed as a browser extension (Chrome/Firefox/Edge) plus Python and Node.js client li… | 87 | 10997 | active |
| Mautic Mautic is the world's largest open source marketing automation platform, offering email marketing, campaign building, lead nurturing, landi… | 99 | 10822 | stable |
| nexmoe/VidBee VidBee is a free, open-source Electron desktop app for downloading video and audio from 1000+ sites (via yt-dlp) and importing local media.… | 80 | 10689 | active |
| averygan/reclip ReClip is a self-hosted web application for downloading videos and audio from 1000+ sites (YouTube, TikTok, Instagram, etc.) using yt-dlp a… | 52 | 10681 | active |
| simonlin1212/a-stock-data A self-contained Skill file (structured Markdown with embedded Python) that aggregates China A-share market data from 19 sources into 54 re… | 80 | 10617 | active |
| StarTrail-org/PixelRAG PixelRAG is a Python library and hosted service for visual retrieval-augmented generation: it renders web pages and documents into screensh… | 77 | 10320 | active |
| SuperSplat SuperSplat is a free, open-source browser-based editor for inspecting, editing, optimizing, and publishing 3D Gaussian Splats. It is built … | 88 | 10256 | active |
| XIU2/UserScript A collection of personal Tampermonkey/GreasyFork userscripts by X.I.U, including GitHub high-speed download enhancements, AutoPager infinit… | 74 | 10069 | active |
| companion-inc/feynman Feynman is an open-source AI research agent that reads papers, searches the web, writes cited research drafts, plans experiments, and ranks… | 78 | 9938 | active |
| Miniflux Miniflux is a minimalist, opinionated, self-hosted RSS/Atom/JSON Feed reader written in Go with PostgreSQL as its only dependency. It empha… | 100 | 9812 | stable |
| xberg-io/xberg Xberg is a polyglot document intelligence framework with a Rust core that extracts text, metadata, tables, images, and structured data from… | 88 | 9437 | active |
| langchain-ai/local-deep-researcher A fully local web research assistant that runs any LLM via Ollama or LMStudio to iteratively search the web, summarize findings, identify k… | 67 | 9378 | active |
| LearningCircuit/local-deep-research A self-hosted AI research assistant that performs deep, iterative multi-source research using local or cloud LLMs (Ollama, llama.cpp, OpenA… | 82 | 9359 | active |
| miurla/morphic Morphic is an open-source AI-powered search engine that generates grounded, cited answers with a generative UI that renders rich inline com… | 93 | 9181 | active |
| open-pencil/open-pencil OpenPencil is an open-source, AI-native design editor that opens and writes native Figma (.fig) files, with built-in AI chat that can creat… | 83 | 9153 | active |
| nexu-io/html-anything HTML Anything is a local-first, agentic HTML editor that drives coding-agent CLIs you already have authenticated (Claude Code, Cursor, Code… | 57 | 8934 | active |
| yogeshojha/rengine reNgine is a self-hosted automated web reconnaissance and vulnerability scanning framework with configurable recon engines, data correlatio… | 66 | 8869 | active |
| firerpa/lamda FIRERPA (lamda) is an all-in-one Android device control platform whose server runs directly on the device (root or non-root) and exposes 16… | 97 | 8593 | active |
| TeamWiseFlow/xiaobei Xiaobei is an open-source multi-agent system that automates social media marketing and customer acquisition for solo entrepreneurs and smal… | 89 | 8574 | active |
| chengaopan/AutoMergePublicNodes An automated aggregator that scrapes and merges publicly available proxy nodes (SS/SSR/V2Ray/Trojan/Clash) from the internet and republishe… | 72 | 8243 | active |
| six2dez/reconftw reconFTW is an open-source (MIT) automated reconnaissance framework written in Shell that orchestrates 80+ security tools to perform full r… | 82 | 8155 | active |
| RayVentura/ShortGPT ShortGPT is a Python framework that automates AI-driven video content creation, combining LLM script generation, voiceover synthesis, asset… | 31 | 7973 | active |
| ValueCell-ai/ClawX ClawX is an open-source Electron desktop application that provides a graphical interface for OpenClaw AI agents, replacing terminal-based A… | 78 | 7607 | active |
| zhinianboke/xianyu-auto-reply A self-hosted multi-account automation system for the Xianyu (Goofish) second-hand marketplace, built with Python/FastAPI, React, MySQL, Re… | 62 | 7399 | active |
| Snowplow Snowplow is a Customer Data Infrastructure platform that collects, validates, enriches, and streams event-level behavioral data from web, m… | 59 | 7034 | stable |
| urbanadventurer/WhatWeb WhatWeb is a command-line web scanner that identifies the technologies powering websites, including CMSs, web servers, JavaScript libraries… | 70 | 6869 | stable |
| zarazhangrui/follow-builders An AI agent skill that tracks top AI builders on X/Twitter, YouTube podcasts, and AI company blogs, then remixes their content into digesti… | 59 | 6828 | active |
| Mobile Next Automation Platform An open-source Model Context Protocol (MCP) server that lets AI agents and LLMs control iOS and Android devices—emulators, simulators, and … | 84 | 6800 | active |
| steipete/summarize Summarize is a Node.js CLI and Chrome/Firefox extension that extracts clean text from web pages, PDFs, YouTube videos, podcasts, and audio/… | 84 | 6729 | active |
| fkling/astexplorer A web-based tool for exploring the abstract syntax trees (ASTs) generated by dozens of parsers for JavaScript, CSS, HTML, GraphQL, YAML, an… | 23 | 6546 | active |
| lavague-ai/LaVague LaVague is an open-source Python framework for building AI Web Agents powered by a Large Action Model. It combines a World Model that inter… | 26 | 6392 | active |
| microsoft/fara Fara1.5 is a family of open-weight computer use agent (CUA) models from Microsoft Research, released at 4B, 9B, and 27B scales and built on… | 60 | 6221 | active |
| agent-infra/sandbox AIO Sandbox is an all-in-one Docker container environment for AI agents that bundles a browser (with VNC), shell terminal, filesystem, VSCo… | 77 | 6068 | active |
| TideDra/zotero-arxiv-daily A Python tool deployed as a GitHub Action that recommends new arXiv (and bioRxiv/medRxiv/chemRxiv) papers daily based on the contents of yo… | 79 | 6002 | active |
| w37fhy/QuantumultX A curated collection of ready-made ('lazy') configuration files and synced community scripts for proxy tools like Quantumult X, Surge, Shad… | 60 | 5916 | active |
| OSMnx OSMnx is a Python library for downloading, modeling, analyzing, and visualizing street networks and other geospatial features from OpenStre… | 71 | 5875 | stable |
| AidLux (AidLearning) AidLux (originally AidLearning) is an AIoT development platform that runs a native Ubuntu Linux environment with GUI, deep learning tooling… | 63 | 5812 | active |
| Miru Miru is a free, open-source, multi-platform media entertainment app built with Flutter for watching videos, reading comics/manga, and readi… | 42 | 5769 | active |
| Tampermonkey/tampermonkey Tampermonkey is the most popular userscript manager, available as a browser extension for Chrome, Edge, Safari, Opera, and Firefox. It lets… | 34 | 5765 | active |
| elceef/dnstwist dnstwist is a Python command-line tool that generates permutations of a domain name (typos, homoglyphs, IDN tricks) and checks which ones r… | 26 | 5758 | active |
| OWASP/Nettacker OWASP Nettacker is a Python-based automated penetration testing and information-gathering framework for reconnaissance, vulnerability scann… | 84 | 5616 | active |
| biliup Biliup is a toolkit for automatically recording livestreams (bilibili, Douyu, Huya, Douyin, etc.), downloading videos from YouTube/Twitch, … | 94 | 5471 | active |
| Go Music DL A Go-based music search and download tool aggregating 10+ platforms (NetEase, QQ Music, Kugou, Bilibili, etc.) with lossless FLAC support, … | 83 | 5433 | active |
| kaifcodec/user-scanner A 2-in-1 Python CLI OSINT suite that performs email and username intelligence across 440+ scan vectors (170+ email sites, 270+ username pla… | 83 | 5425 | active |
| DeepResearch A TypeScript agent that iteratively searches the web, reads webpages, and reasons with an LLM (Gemini, OpenAI, or local) until it finds a c… | 55 | 5237 | active |
| QLHazyCoder/FlowPilot FlowPilot is a Chrome sidebar extension for batch-automating ChatGPT/OpenAI, Kiro, and Grok account registration, OAuth authorization, and … | 71 | 5179 | active |
| liam-hq/liam Liam ERD is an open-source tool that automatically generates beautiful, interactive entity-relationship diagrams from database schema files… | 70 | 5148 | active |
| techchipnet/CamPhish CamPhish is a bash-based penetration-testing tool that hosts a fake webpage on a built-in PHP server and exposes it via ngrok or CloudFlare… | 37 | 5147 | active |
| developersdigest/llm-answer-engine A Perplexity-inspired AI answer engine built with Next.js, the Vercel AI SDK, LangChain.JS, Groq/Mixtral, Brave Search, Serper, and OpenAI.… | 58 | 5035 | active |
| Bitcore Bitcore is a full-stack monorepo from BitPay providing infrastructure for building Bitcoin and blockchain-based applications, including a b… | 67 | 4982 | active |
| qixing-jk/all-api-hub All API Hub is an open-source browser extension (Chrome, Edge, Firefox, Safari) that acts as an all-in-one manager for AI relay/proxy site … | 80 | 4919 | active |
| imroc/req Req is a simple yet powerful Go HTTP client library with chainable APIs, supporting HTTP/1.1, HTTP/2, and HTTP/3. It offers built-in debugg… | 94 | 4866 | active |
| capricorn86/happy-dom Happy DOM is a JavaScript implementation of a web browser without its graphical user interface, providing a DOM implementation for Node.js … | 95 | 4699 | active |
| mcncarl/yichen-skills A collection of Claude Code and Codex skills for content creators, covering writing workflows, X Articles publishing, WeChat/WeCom data cap… | 69 | 4661 | active |
| TencentCloudADP/youtu-agent Youtu-Agent is a Python framework for building, running, and evaluating autonomous LLM agents, supporting ReAct-style single agents and pla… | 52 | 4622 | active |
| cyrus-and/chrome-remote-interface A Node.js library providing a JavaScript interface to the Chrome Debugging Protocol (CDP). It abstracts CDP domains into commands and event… | 58 | 4556 | stable |
| yihong0618/running_page A self-hosted personal running homepage that syncs workout data from services like Strava, Garmin, Nike Run Club, and GPX files, then visua… | 77 | 4540 | active |
| kaixindelele/ChatPaper ChatPaper is a Python tool that uses ChatGPT to summarize arXiv papers, translate full PDFs, polish academic writing, and simulate peer rev… | 59 | 19943 | maintenance |
| jonaslejon/malicious-pdf A Python CLI tool that generates 67 malicious PDF test files embedding callbacks for SSRF, XSS, XXE, NTLM credential theft, and data exfilt… | 90 | 4483 | active |
| gptme gptme is a free, open-source personal AI agent that runs in the terminal, equipped with local tools for shell execution, code writing, file… | 98 | 4457 | active |
| magcius/noclip.website noclip.website is a browser-based application that lets users explore and fly through video game levels rendered from reverse-engineered mo… | 77 | 4288 | active |
| hipcityreg/situation-monitor A real-time dashboard application for monitoring global news, financial markets, and geopolitical events. It aggregates live information st… | 40 | 4213 | active |
| KuekHaoYang/KVideo KVideo is a modern video aggregation and playback platform built on Next.js 16, React 19, and Tailwind CSS v4, featuring a distinctive 'Liq… | 58 | 4189 | active |
| magnitudedev/browser-agent Magnitude is an open-source, vision-first AI browser agent written in TypeScript that controls the browser through natural language command… | 56 | 4134 | active |
| groupultra/telegram-search A self-hostable tool that exports, backs up, and indexes Telegram chat history into a database (PGlite or PostgreSQL) with tokenization and… | 84 | 4125 | active |
| zhom/donutbrowser Donut Browser is an open-source anti-detect desktop browser for macOS, Windows, and Linux that creates unlimited isolated browser profiles,… | 83 | 4088 | active |