resource: web-scraping
198 resources, primary matches first, then adoption-weighted; health v2 shown.
| Resource | Health v2 | Stars | Maturity |
|---|---|---|---|
| luzhisheng/js_reverse A curated collection of study notes and tool links for learning web scraping reverse engineering, covering JavaScript deobfuscation, app re… | 76 | 1319 | active |
| Kr1s77/Python-crawler-tutorial-starts-from-zero A Chinese-language tutorial repository teaching Python web scraping from zero, covering requests, data extraction, JS reverse engineering, … | 32 | 4613 | maintenance |
| quemsah/awesome-claude-plugins An automatically curated awesome-list directory of Claude Code plugins and their adoption metrics, built with n8n workflows that scan GitHu… | 61 | 1264 | active |
| ZGQ-inc/source A large curated collection of importable sources for Chinese reading and media apps, including ~28,000 book sources for Legado (开源阅读), RSS … | 41 | 1257 | active |
| ayoubfathi/leaky-paths A curated wordlist of special web paths linked to sensitive APIs, devops internals, framework configs, and known misconfigurations. It is d… | 66 | 1193 | active |
| tb0hdan/domains A public dataset of billions of sorted Internet domains, described as the world's largest, built by processing petabytes of crawl and DNS o… | 68 | 1153 | active |
| 51bitquant/bitquant Companion code repository for 51bitquant's YouTube/Bilibili video series on cryptocurrency quantitative trading in Python, using CCXT to fe… | 30 | 1153 | active |
| xingag/spider_python A collection of Python web scraping examples and tutorials covering requests, xpath, BeautifulSoup, regex, multithreading, Selenium, Scrapy… | 75 | 1152 | active |
| needleworm/bhban_rpa Example code repository for the Korean book '6개월 치 업무를 하루 만에 끝내는 업무 자동화' (2020), teaching office automation with Python to complete beginne… | 77 | 1149 | active |
| MinorJerry/WebVoyager WebVoyager is the official code and dataset for a research paper on an end-to-end web agent powered by large multimodal models that complet… | 26 | 1122 | active |
| cipher387/python-for-OSINT-21-days A 21-day beginner course teaching Python for open-source intelligence (OSINT) work, provided as a PDF/Notion curriculum with sample code fi… | 58 | 1120 | active |
| leomaurodesenv/game-datasets A curated awesome-list of game datasets, APIs, tools, and resources for applying artificial intelligence and data mining to digital games. … | 75 | 1109 | active |
| HelloWorld521/Java A collection of Java practice projects covering Spring Boot with Shiro, a seckill system, a servlet-based web server, JSP chatroom, library… | 32 | 3786 | maintenance |
| serpapi/awesome-seo-tools A curated awesome-list of SEO (Search Engine Optimization) tools maintained by SerpApi, organized into categories like keyword research, ba… | 58 | 1078 | active |
| EBazarov/nsfw_data_source_urls A curated collection of text files listing over 1.5 million image URLs across 159 categories, intended for downloading and building NSFW im… | 32 | 3577 | maintenance |
| howie6879/weekly A personal Chinese-language tech weekly newsletter (老胡的信息周刊) that curates valuable projects, software, tutorials, and websites in the compu… | 77 | 1027 | active |
| Rivens7/Livelist A curated IPTV live stream source list focused on IPv6, auto-updated and synced from popular Chinese IPTV projects like fanmingming/live an… | 60 | 1022 | active |
| wkunzhi/Python3-Spider A collection of dozens of Python 3 web scraping examples demonstrating simulated logins, slider captcha handling, font encryption, and logi… | 32 | 3386 | maintenance |
| geekan/scrapy-examples A collection of example Scrapy spiders demonstrating web scraping patterns for sites like Douban, LinkedIn, Zhihu, Alexa, and GitHub. It in… | 32 | 3253 | maintenance |
| wecatch/china_regions A dataset of China's administrative divisions (province, city, county, town, village) scraped from the national statistics bureau standard,… | 23 | 3155 | maintenance |
| checkly/puppeteer-examples A collection of practical Puppeteer example scripts for running Headless Chrome from Node.js, covering basics like screenshots and form han… | 10 | 3041 | maintenance |
| transitive-bullshit/awesome-puppeteer A curated awesome-list of Puppeteer resources for controlling headless Chrome or Chromium via the DevTools Protocol. It links to official d… | 32 | 2572 | maintenance |
| codingforentrepreneurs/30-Days-of-Python A 30-day Python tutorial series by Coding for Entrepreneurs, with this repository containing supporting code and materials for the video co… | 32 | 2504 | maintenance |
| Python3WebSpider/Python3WebSpider Source code repository for the Chinese book 'Python3 Web Spider Development in Practice' by Cui Qingcai. It contains example code covering … | 32 | 2434 | maintenance |
| emadehsan/thal A starter guide and example code repository for web scraping using Puppeteer and Chrome Headless in Node.js. It accompanies a Medium articl… | 32 | 2360 | maintenance |
| lemonhu/stock-knowledge-graph A small educational project that builds a securities knowledge graph in Neo4j from public web data, including company executives and stock … | 32 | 2171 | maintenance |
| realpython/python-scripts A curated collection of small, standalone Python scripts covering tasks like web crawling, file manipulation, JSON/CSV conversion, Git help… | 32 | 2159 | maintenance |
| chainfeeds/RSSAggregatorforWeb3 A curated collection of 600+ RSS feeds covering Web3, crypto, and blockchain content, distributed as an OPML file for import into any RSS r… | 23 | 2146 | maintenance |
| DXY-COVID-19 A time-series data warehouse of COVID-19 (2019-nCoV) infection statistics for China, scraped from Dingxiangyuan (DXY) and published as CSV/… | 10 | 1971 | maintenance |
| Ehco1996/Python-crawler A systematic, from-scratch tutorial repository for learning how to write Python web crawlers, with example code for BeautifulSoup, Scrapy, … | 32 | 1915 | maintenance |
| librauee/Reptile A Python3 web scraping learning repository containing dozens of hands-on crawler examples targeting popular Chinese websites like Maoyan, D… | 32 | 1749 | maintenance |
| uo-ec607/lectures A collection of open lecture notes for EC 607, a graduate course on data science for economists at the University of Oregon. The materials … | 32 | 1671 | maintenance |
| lightswitch05/hosts A collection of regularly updated hosts files for blocking ads, trackers, Facebook services, AMP pages, and other unwanted domains. The lis… | 10 | 1527 | maintenance |
| JonasCz/How-To-Prevent-Scraping A comprehensive written guide on techniques for preventing or hindering website scraping, expanded from a Stack Overflow answer. It explain… | 32 | 1517 | maintenance |
| danielmiessler/RobotsDisallowed A curated dataset of robots.txt disallowed directories harvested from the top 100K websites (Alexa/Majestic), useful for content discovery … | 32 | 1490 | maintenance |
| kkdai/project52 A collection of 52 small Go projects written one per week over a year as a personal learning challenge. It serves as example code covering … | 32 | 1470 | maintenance |
| tcc0lin/Review_Reverse A collection of JavaScript reverse engineering case studies and tutorials covering how popular Chinese websites (Taobao, Zhihu, Toutiao, Ba… | 32 | 1429 | maintenance |
| beiliangshizi/China_House A curated Chinese-language collection of resources, guides, and tools about buying and renting houses in China, organized by city (Beijing,… | 32 | 1417 | maintenance |
| scrapy/quotesbot A sample Scrapy project that scrapes quotes, authors, and tags from quotes.toscrape.com using two spiders (CSS selectors and XPath). It is … | 32 | 1360 | maintenance |
| daolf/Most-recommended-programming-books A curated dataset and article listing the 25 most recommended programming books of all time, compiled by aggregating and analyzing roughly … | 32 | 1352 | maintenance |
| KaiDMML/FakeNewsNet FakeNewsNet is a research dataset for fake news detection, with minimal CSV files of news samples from PolitiFact and GossipCop plus Python… | 32 | 1346 | maintenance |
| lovelyyoshino/Bilibili-Live-API A community-maintained collection of API documentation for Bilibili's live streaming and anime services, reverse-engineered from the platfo… | 75 | 1337 | maintenance |
| SimmerChan/KG-demo-for-movie A tutorial demo that builds a movie knowledge graph from scratch (MySQL → D2RQ → RDF → Jena Fuseki) and implements a simple KBQA (knowledge… | 32 | 1329 | maintenance |
| ityard/python-fxxk-spider A curated collection of free, open-source Python web crawler/spider projects covering sites like Weibo, Douban, JD, Zhihu, and Douyin. It s… | 32 | 1274 | maintenance |
| alexattia/Data-Science-Projects A collection of personal data science and machine learning projects built around Kaggle and HackerRank challenges, written in Python and Ju… | 32 | 1187 | maintenance |
| rogerzhu/MNWeeklyCategory A categorized collection of over 10,000 curated links from 327+ issues of the Chinese 'Manong Weekly' (码农周刊) newsletter, organized into mar… | 32 | 1161 | maintenance |
| ryfeus/lambda-packs A collection of precompiled Python packages bundled as ZIP files ready to upload to AWS Lambda, covering libraries like TensorFlow, PyTorch… | 32 | 1116 | maintenance |
| MrLevo520/Mini-Python-Project A collection of small, fun Python mini-projects with tutorials aimed at beginners/freshmen, written in Jupyter Notebooks. Projects cover we… | 32 | 1075 | maintenance |
| LoseNine/Crack-JS-Spider A curated collection of JavaScript reverse-engineering solutions for bypassing anti-scraping encryption parameters on popular Chinese websi… | 32 | 1034 | maintenance |
| alanchn31/Data-Engineering-Projects A collection of personal data engineering projects completed as part of Udacity's Data Engineering Nanodegree, covering ETL into Postgres, … | 32 | 1030 | maintenance |
| iphysresearch/TOP250movie_douban A Jupyter Notebook-based educational project that scrapes Douban Top 250 movie short reviews with Scrapy, then performs data cleaning, stat… | 32 | 1022 | maintenance |
| majacinka/crewai-experiments A collection of experiments with the CrewAI multi-agent framework, testing local models via Ollama and API models like GPT-4 and Gemini Pro… | 26 | 1017 | experimental |
| SocialSisterYi/bilibili-API-collect A community-collected documentation of Bilibili's unofficial/internal API endpoints, parameters, and authentication logic. It was permanent… | 10 | 20235 | abandoned |
| Mining the Social Web The official code companion repository for the book 'Mining the Social Web, 2nd Edition' (O'Reilly, 2013), containing Jupyter notebook exam… | 32 | 2878 | abandoned |
| feeddd/feeds A community-maintained collection of free RSS/Atom/JSON feeds for WeChat official accounts, generated via Hamibot Android automation script… | 32 | 2098 | abandoned |
| scrapy/dirbot A deprecated educational Scrapy example project that scraped the public DMOZ web directory. It was retired in March 2017 when dmoz.org shut… | 32 | 1628 | abandoned |
| AlanChen4/Summer-2024-SWE-Internships A curated, auto-updated list of Summer 2024 software engineering internship postings, maintained by a Python job monitor and linked to the … | 10 | 1362 | abandoned |
| stanfordjournalism/search-script-scrape A collection of 101 real-world web scraping and data-collection exercises in Python 3, created for Stanford's Computational Journalism clas… | 32 | 1234 | abandoned |
| zhentaoo/puppeteer-deep A collection of Puppeteer/Headless Chrome demo scripts in JavaScript showing advanced web scraping, automated UI interactions (auto-posting… | 32 | 1209 | abandoned |
| marchtea/scrapy_doc_chs A Chinese translation of the Scrapy web-scraping framework documentation, hosted on Read the Docs and built with Sphinx. It is a community … | 32 | 1102 | abandoned |
| Asabeneh/30-Days-Of-Python A free, step-by-step 30-day curriculum for learning the Python programming language, covering fundamentals through web development, APIs, d… | 76 | 71964 | stable |
| geekcomputers/Python A large collection of small, standalone Python example scripts written to automate everyday tasks and demonstrate Python concepts to beginn… | 76 | 35292 | active |
| KalyanM45/AI-Project-Gallery A curated gallery of artificial intelligence projects spanning machine learning, deep learning, NLP, computer vision, and generative AI, ea… | 71 | 6404 | active |
| decodingai-magazine/llm-twin-course A free hands-on course (source code plus 12 lessons) that teaches how to build an end-to-end production-ready LLM and RAG system by creatin… | 60 | 4385 | active |
| alsotang/node-lessons A Chinese-language Node.js tutorial course ('Node.js 包教不包会') by alsotang, consisting of 19 hands-on lessons covering Express, web scraping,… | 32 | 16434 | maintenance |
| sanjeed5/awesome-cursor-rules-mdc A curated awesome list of Cursor .mdc rule files, paired with a Python generator that creates MDC rule files from a JSON library catalog. I… | 58 | 3569 | active |
| MorvanZhou/tutorials Companion code repository for the MorvanPython (莫烦Python) tutorial series, covering machine learning, neural networks (TensorFlow, PyTorch,… | 32 | 13010 | maintenance |
| dloss/python-pentest-tools A curated list of Python tools, libraries, and bindings useful for penetration testers, vulnerability researchers, and reverse engineers. I… | 74 | 2883 | active |
| RKiding/Awesome-finance-skills A curated collection of plug-and-play Agent Skills that equip LLM-based AI agents with financial analysis capabilities, including real-time… | 49 | 2812 | active |
| vector4wang/spring-boot-quick A Spring Boot learning hub with 80+ example modules demonstrating real-world integration of frameworks like RabbitMQ, Kafka, MyBatis, JPA, … | 69 | 2788 | active |
| eddwebster/football_analytics A collection of football (soccer) analytics projects, notebooks, and data pipelines by Edd Webster, covering topics like expected goals, xT… | 53 | 2749 | active |
| Tanu-N-Prabhu/Python A collection of Jupyter notebooks and tutorials for learning Python, data science, and machine learning from scratch. It covers topics like… | 86 | 2231 | active |
| badmojr/1Hosts 1Hosts is a set of maintained DNS filter/blocklists that block ads, trackers, and malware domains. It ships in multiple formats (hosts, dom… | 88 | 2209 | active |
| dreddsa5dies/goHackTools A collection of small Go example programs implementing classic hacker and pentester tools such as port scanners, password bruteforcers, DNS… | 74 | 2187 | active |
| PuddinCat/BestClash A free Clash/Mihomo proxy subscription that automatically scrapes public proxy sources from the web, speed-tests them, and publishes an upd… | 64 | 1917 | active |
| midudev/curso-node-js A free, practical Node.js course taught live on Twitch by midudev, with recorded video lessons on YouTube and course materials hosted on Gi… | 63 | 1683 | active |
| soruly/awesome-acg A curated awesome-list of tools, libraries, and projects related to anime, comics, games, light novels, visual novels, vocaloid, and touhou… | 74 | 1469 | active |
| Pluviobyte/rnskill A curated collection of 56 AI Agent Skills (SKILL.md packages) for Codex, Claude Code, and similar agent workflows, split into content-crea… | 67 | 1402 | active |
| modal-labs/modal-examples A curated collection of example programs for Modal, a serverless cloud platform, covering use cases like LLM serving, image generation, spe… | 77 | 1264 | active |
| navilg/media-stack A Docker Compose-based self-hosted media stack bundling Sonarr, Radarr, Prowlarr, qBittorrent, Seerr, Jellyfin, and AI-powered recommendati… | 77 | 1231 | active |
| Show-Me-the-Code/python A community-driven collection of small Python programming exercises ('Show Me the Code'), where contributors submit their own solutions via… | 32 | 4301 | maintenance |
| CodeCutTech/Data-science A curated collection of data science articles, videos, and code examples from CodeCut, covering topics like MLOps, data management, testing… | 10 | 4220 | maintenance |
| skygazer42/DL-Hub DL-Hub is a comprehensive PyTorch deep learning learning repository with 339 lessons across 8 tracks (vision, NLP, GNN, point cloud, genera… | 77 | 1114 | active |
| 4chan/4chan-API Official documentation for 4chan's read-only JSON API, describing endpoints for boards, threads, catalogs, and archives served from a.4cdn.… | 75 | 1109 | active |
| phodal/ideabook An open-source ebook (CC0) by Phodal presenting a collection of hands-on full-stack practice project ideas with step-by-step guides, such a… | 32 | 3760 | maintenance |
| shibing624/python-tutorial A practical Python tutorial repository with Jupyter Notebook examples covering Python basics, advanced features, object-oriented programmin… | 32 | 2470 | maintenance |
| sebgl/htpc-download-box A Docker Compose-based setup that deploys an automated home media server stack combining Sonarr, Radarr, Jackett, NZBGet, Deluge, OpenVPN, … | 32 | 2139 | maintenance |
| cs109/content Official course content for Harvard's CS109 Data Science course, including homework assignments, lecture supplements, and labs as Jupyter n… | 32 | 1793 | maintenance |
| justmarkham/DAT8 Course materials for General Assembly's 2015 Data Science course in Washington, DC, taught by Kevin Markham. It contains Jupyter notebooks … | 32 | 1620 | maintenance |
| HarshCasper/Rotten-Scripts A curated collection of small utility scripts written in Python, JavaScript, Bash, PowerShell, and Go, contributed by the open-source commu… | 32 | 1546 | maintenance |
| ElemeFE/node-practice A Node.js practice tutorial from ElemeFE that teaches advanced Node.js concepts by implementing well-known modules like Promise, co, koa, H… | 32 | 1357 | maintenance |
| simon987/awesome-datahoarding A curated awesome-list of data-hoarding related tools covering web archiving, download utilities, backup, compression, file systems, and lo… | 32 | 1333 | maintenance |
| xuwujing/java-study A personal Java study repository containing example code covering Java basics, JDK 8 features, concurrency, the 23 design patterns, and pop… | 32 | 1306 | maintenance |
| tdamdouni/Pythonista A curated collection of Python scripts written for the Pythonista iOS app, gathered from GitHub forks, the omz:forum, and other coding comm… | 32 | 1164 | maintenance |
| momo0853/kkndme An archived collection of the famous Tianya forum thread 'kkndme聊房' about Chinese real estate and housing investment, compiled into HTML, P… | 32 | 1159 | maintenance |
| 3had0w/Fuzzing-Dicts A curated collection of dictionaries and wordlists for web security testing, including payloads for fuzzing, directory brute-forcing, and v… | 32 | 1028 | maintenance |
| h2y/Shadowrocket-ADBlock-Rules A collection of auto-generated proxy rule files for the Shadowrocket iOS app, defining which sites go through a proxy versus direct connect… | 10 | 16696 | abandoned |
| tongtzeho/PKUCourse A curated collection of undergraduate and graduate computer science course projects from Peking University's EECS department, spanning 2011… | 32 | 2551 | abandoned |
← prev page 2 / 2