brianway/webporter

Health v2 · maintenance only

Flags: no_license

Adoption not part of the score

2766 stars · 847 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

webporter is a Java crawler application built on the webmagic framework that demonstrates a complete pipeline of data crawling, persistence, and visualization. It ships a Zhihu user-data collector example with modules for core logic, Elasticsearch bulk import, and site-specific collection configured via JSON.

Use cases

• scrape zhihu user profiles into elasticsearch
• learn how to build a java crawler with webmagic
• bulk import crawled json data into elasticsearch
• visualize scraped data with kibana aggregations
• example of a full crawl-store-analyze pipeline
• configure a crawler without changing source code

When to choose

• you want a worked Java example of crawling plus Elasticsearch ingestion
• you need to collect Zhihu user data for analysis or learning
• you prefer JSON configuration over code changes

When to avoid

• you need a production-grade crawler - use webmagic or scrapy instead
• you need a general-purpose scraping framework rather than a sample application
• you need an actively maintained project with a clear license

Facets

application · maturity maintenance

web-scraping search-engine data-visualization etl crawlers data-science developer-tools jvm cli zhihu webmagic elasticsearch kibana crawler-example json-config multi-threaded search docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="brianway/webporter")