# zhegexiaohuozi/SeimiCrawler

一个简单、敏捷、分布式的支持SpringBoot的Java爬虫框架;An agile, distributed crawler framework.

Repository: https://github.com/zhegexiaohuozi/SeimiCrawler
Canonical: https://ross.abutalabs.com/products/seimicrawler
Homepage: http://seimicrawler.org
Language: Java
License: Apache-2.0
License Family: permissive
Last push: 2026-07-29T05:42:38+00:00

## Health v2 (maintenance only)
Score: 85/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 95, release rhythm 63, longevity 100
- inputs: {"age_days": 3953, "days_push": 35, "days_rel": 35, "gap_med": null, "n_releases_24m": 1}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1990, forks 683 (observed 2026-08-28T04:06:03.078271+00:00)

## What it is
SeimiCrawler is an agile, standalone, distributed Java crawler framework inspired by Python's Scrapy, with deep Spring Boot integration and Redisson-based distributed queues. It uses JsoupXpath for HTML parsing by default and integrates SeimiRender (a Chromium-based rendering service) to handle JavaScript-rendered and dynamic pages.

## Use cases
- build a distributed web crawler in Java
- scrape websites with Spring Boot
- crawl JavaScript-rendered SPA pages
- extract data from HTML using XPath
- run large-scale data collection with deduplication via Redis
- migrate a Scrapy-style pipeline to the JVM

## When to choose
- you need a Scrapy-like crawler framework on the JVM
- you want distributed crawling backed by Redis/Redisson with BloomFilter dedup
- your project already uses Spring Boot
- you need to scrape dynamic pages via a headless-browser rendering service

## When to avoid
- you need a simple one-off script where a lightweight HTTP client suffices
- your stack is Python (Scrapy may fit better)
- you want a no-code scraping tool rather than a code-first framework
- you cannot run Java 8+ or a Redis instance for distributed mode

## Facets
- artifact type: framework
- maturity: active
- function: web-scraping, parser, http-client, workflow-automation
- domain: crawlers, web-development
- platform: jvm, cross-platform
- tags: crawler-framework, xpath, distributed-crawling, spring-boot, redisson, scrapy-inspired, jsoup, data-engineering, automation

## Member repositories
- zhegexiaohuozi/SeimiCrawler (main) score 85

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:06:03.078271+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T03:02:46.777310+00:00, confidence not recorded.
  - readme: https://github.com/zhegexiaohuozi/SeimiCrawler (fetched 2026-08-28T04:06:03.078271+00:00, sha dd1015efd537)
  - homepage: http://seimicrawler.org (fetched 2026-08-29T10:42:52.027398+00:00, sha 1f5bf81be76b)
- Data as of 2026-08-30T08:39:29.467469+00:00.