# andeya/pholcus

Pholcus is a distributed high-concurrency crawler software written in pure golang

Repository: https://github.com/andeya/pholcus
Canonical: https://ross.abutalabs.com/products/pholcus
Language: Go
License: Apache-2.0
License Family: permissive
Topics: spider, crowler
Last push: 2026-08-24T03:24:17+00:00

## Health v2 (maintenance only)
Score: 90/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 99, release rhythm 73, longevity 100
- inputs: {"age_days": 4079, "days_push": 9, "days_rel": 183, "gap_med": 1, "n_releases_24m": 2}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 7577, forks 1666 (observed 2026-08-28T04:10:01.734338+00:00)

## What it is
Pholcus is a distributed, high-concurrency web crawler framework written in pure Go. It supports standalone, server, and client modes with Web UI, GUI, and command-line interfaces, plus pluggable static (Go) and dynamic (JS/XML) crawl rules.

## Use cases
- scrape websites at high concurrency in Go
- run a distributed crawling cluster with task distribution
- crawl JavaScript-heavy pages with a headless Chrome downloader
- export scraped data to MySQL, MongoDB, Kafka, CSV, or Excel
- rotate proxy IPs and manage cookies to avoid blocking
- resume interrupted crawl jobs from persisted state

## When to choose
- you need a full-featured Go crawler with distributed task scheduling out of the box
- you want both static and hot-reloadable dynamic crawl rules
- you need multiple output targets like databases, message queues, and files
- you want built-in proxy rotation, retry, deduplication, and checkpointing

## When to avoid
- you need a lightweight single-purpose scraper rather than a full framework
- your project is not in Go and you cannot use Go rules
- you require a maintained ecosystem with frequent updates and large community support
- legal restrictions in your jurisdiction prohibit web scraping for your use case

## Facets
- artifact type: framework
- maturity: active
- function: web-scraping, http-client, caching, etl
- domain: crawlers, web-development
- platform: go, cross-platform, cli, windows
- tags: distributed-crawler, high-concurrency, spider-framework, headless-browser, proxy-pool, web-ui, gui, hot-reload-rules, data-engineering, automation, web-server, linux, macos

## Member repositories
- andeya/pholcus (main) score 90

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:10:01.734338+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T17:36:55.126561+00:00, confidence not recorded.
  - readme: https://github.com/andeya/pholcus (fetched 2026-08-28T04:10:01.734338+00:00, sha 6ecebd597fd1)
- Data as of 2026-08-30T08:39:29.467469+00:00.