code4craft/webmagic
A scalable web crawler framework for Java. observed · 2026-08-28
Health v2 · maintenance only
60/100
- Activity 58
- Release rhythm 40
- Longevity 100
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-02. Adoption (stars, forks) is never an input.
- gap_med: 0.0
- age_days: 4880
- days_rel: 569
- days_push: 256
- n_releases_24m: 7
Adoption not part of the score
11678 stars · 4106 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded
WebMagic is a scalable web crawler framework for Java covering the full crawl lifecycle: downloading, URL management, content extraction (XPath/CSS/regex/JsonPath), and persistence. It offers a simple chainable API plus annotation-driven POJO crawlers, with multi-threading and Redis-based distributed support.
Use cases
- scrape product prices from e-commerce sites
- crawl github repository metadata
- extract structured data from html pages
- build a distributed crawler with redis
- collect news articles into a database
- parse web pages with xpath and css selectors
When to choose
- you need a JVM-based crawler with a simple, extensible API
- you want annotation-driven extraction into POJOs
- you need multi-threaded or Redis-distributed crawling
- you want pluggable pipelines to console, file, MySQL, or Redis
When to avoid
- you need a headless browser for JavaScript-heavy sites
- your stack is Python (Scrapy) or Node.js
- you only need a one-off simple HTTP fetch
- you need a maintained GUI scraping tool
Facets
framework · maturity active
web-scraping parser http-client etl crawlers web-development developer-tools jvm cross-platform crawler scraping xpath distributed-crawling annotation-driven data-engineering
5 sources
- readme: https://github.com/code4craft/webmagic · fetched 2026-08-28 · ba1f79264805
- homepage: http://webmagic.io/ · fetched 2026-08-29 · 4494c3c4d1dc
- site_page: http://webmagic.io/archive/docs/0.6.1/en/index.html · fetched 2026-08-29 · 256c6ea19934
- site_page: http://webmagic.io/apidocs/index.html · fetched 2026-08-29 · d13e3138b44e
- site_page: http://webmagic.io/archive/docs/0.6.1/zh/index.html · fetched 2026-08-29 · 064d5208ee6b
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| code4craft/webmagic | main | 60 |
For agents
markdown · JSON · MCP: product_card(name="code4craft/webmagic")
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem