CrawlScript/WebCollector
WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes. observed · 2026-08-28
Health v2 · maintenance only
62/100
- Activity 66
- Release rhythm 35
- Longevity 100
Flags: no_releases
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-02. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 4435
- days_rel: n/a
- days_push: 204
- n_releases_24m: 0
Adoption not part of the score
3083 stars · 1427 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded
WebCollector is an open-source Java web crawler framework that provides simple interfaces for building multi-threaded web crawlers quickly. It supports features like cookies, POST requests, proxies, and Selenium-based JavaScript rendering.
Use cases
- crawl websites to collect pages
- build a multi-threaded web crawler in Java
- scrape news articles from the web
- crawl JavaScript-rendered pages with Selenium
- collect data for building a search index
- crawl sites through rotating proxies
When to choose
- you need a quick-to-setup crawler on the JVM
- you want a lightweight Java alternative to Scrapy-style crawling
- you need multi-threaded crawling with proxy and cookie support
When to avoid
- you need a distributed, large-scale crawling platform
- your stack is Python or Node.js rather than Java
- you need heavy-duty headless-browser automation beyond simple rendering
Facets
framework · maturity maintenance
web-scraping http-client parser crawlers web-development jvm cross-platform web-crawler multi-threaded selenium java-framework search-engine-crawling data-engineering
2 sources
- readme: https://github.com/CrawlScript/WebCollector · fetched 2026-08-28 · 4be37c3fc3cb
- homepage: https://github.com/CrawlScript/WebCollector · fetched 2026-08-29 · 6f08d5e09791
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| CrawlScript/WebCollector | main | 62 |
For agents
markdown · JSON · MCP: product_card(name="CrawlScript/WebCollector")
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem