zhegexiaohuozi/SeimiCrawler

Health v2 · maintenance only

Adoption not part of the score

1990 stars · 683 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

SeimiCrawler is an agile, standalone, distributed Java crawler framework inspired by Python's Scrapy, with deep Spring Boot integration and Redisson-based distributed queues. It uses JsoupXpath for HTML parsing by default and integrates SeimiRender (a Chromium-based rendering service) to handle JavaScript-rendered and dynamic pages.

Use cases

• build a distributed web crawler in Java
• scrape websites with Spring Boot
• crawl JavaScript-rendered SPA pages
• extract data from HTML using XPath
• run large-scale data collection with deduplication via Redis
• migrate a Scrapy-style pipeline to the JVM

When to choose

• you need a Scrapy-like crawler framework on the JVM
• you want distributed crawling backed by Redis/Redisson with BloomFilter dedup
• your project already uses Spring Boot
• you need to scrape dynamic pages via a headless-browser rendering service

When to avoid

• you need a simple one-off script where a lightweight HTTP client suffices
• your stack is Python (Scrapy may fit better)
• you want a no-code scraping tool rather than a code-first framework
• you cannot run Java 8+ or a Redis instance for distributed mode

Facets

framework · maturity active

web-scraping parser http-client workflow-automation crawlers web-development jvm cross-platform crawler-framework xpath distributed-crawling spring-boot redisson scrapy-inspired jsoup data-engineering automation

Member repositories

For agents

markdown · JSON · MCP: product_card(name="zhegexiaohuozi/SeimiCrawler")