# jaeles-project/gospider

Gospider - Fast web spider written in Go

Repository: https://github.com/jaeles-project/gospider
Canonical: https://ross.abutalabs.com/products/gospider
Language: Go
License: MIT
License Family: permissive
Topics: spider, crawler, go, gospider, bugbounty
Last push: 2024-04-21T08:12:13+00:00

## Health v2 (maintenance only)
Score: 23/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 0, release rhythm 8, longevity 100
- inputs: {"age_days": 2415, "days_push": 864, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 2993, forks 333 (observed 2026-08-28T04:07:36.175007+00:00)

## What it is
GoSpider is a fast web spider/crawler written in Go that crawls sites in parallel and extracts URLs from sitemaps, robots.txt, JavaScript files, and third-party archives like the Wayback Machine and Common Crawl. It is designed for reconnaissance workflows, especially in bug bounty hunting, with grep-friendly output.

## Use cases
- crawl a website and extract all URLs
- find subdomains and AWS S3 buckets in page sources
- pull archived URLs from the Wayback Machine and Common Crawl
- discover links hidden in JavaScript files
- enumerate sitemap.xml and robots.txt entries
- run recon for bug bounty targets
- crawl multiple sites in parallel with custom headers and proxies

## When to choose
- you need a fast, parallel CLI crawler for reconnaissance
- you want URL discovery from JS files and third-party archives in one tool
- you need grep-friendly output to pipe into other recon tools

## When to avoid
- you need a full-featured scraping framework with rendering or data extraction pipelines
- you want a polite, rate-limited crawler for production site indexing
- you need a maintained library API rather than a standalone CLI

## Facets
- artifact type: cli-tool
- maturity: maintenance
- function: web-scraping, cli, parser
- domain: security, crawlers, developer-tools
- platform: cli, windows, go
- tags: bug-bounty, recon, osint, sitemap, wayback-machine, link-finder, subdomain-discovery, command-line, linux, macos, docker

## Member repositories
- jaeles-project/gospider (main) score 23

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:07:36.175007+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T07:30:51.522466+00:00, confidence not recorded.
  - readme: https://github.com/jaeles-project/gospider (fetched 2026-08-28T04:07:36.175007+00:00, sha cb7e315ef496)
- Data as of 2026-08-30T08:39:29.467469+00:00.