jaeles-project/gospider
Gospider - Fast web spider written in Go observed · 2026-08-28
Health v2 · maintenance only
23/100
- Activity 0
- Release rhythm 8
- Longevity 100
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-02. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 2415
- days_rel: n/a
- days_push: 864
- n_releases_24m: 0
Adoption not part of the score
2993 stars · 333 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded
GoSpider is a fast web spider/crawler written in Go that crawls sites in parallel and extracts URLs from sitemaps, robots.txt, JavaScript files, and third-party archives like the Wayback Machine and Common Crawl. It is designed for reconnaissance workflows, especially in bug bounty hunting, with grep-friendly output.
Use cases
- crawl a website and extract all URLs
- find subdomains and AWS S3 buckets in page sources
- pull archived URLs from the Wayback Machine and Common Crawl
- discover links hidden in JavaScript files
- enumerate sitemap.xml and robots.txt entries
- run recon for bug bounty targets
- crawl multiple sites in parallel with custom headers and proxies
When to choose
- you need a fast, parallel CLI crawler for reconnaissance
- you want URL discovery from JS files and third-party archives in one tool
- you need grep-friendly output to pipe into other recon tools
When to avoid
- you need a full-featured scraping framework with rendering or data extraction pipelines
- you want a polite, rate-limited crawler for production site indexing
- you need a maintained library API rather than a standalone CLI
Facets
cli-tool · maturity maintenance
web-scraping cli parser security crawlers developer-tools cli windows go bug-bounty recon osint sitemap wayback-machine link-finder subdomain-discovery command-line linux macos docker
1 source
- readme: https://github.com/jaeles-project/gospider · fetched 2026-08-28 · cb7e315ef496
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| jaeles-project/gospider | main | 23 |
For agents
markdown · JSON · MCP: product_card(name="jaeles-project/gospider")
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem