# hakluke/hakrawler

Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application

Repository: https://github.com/hakluke/hakrawler
Canonical: https://ross.abutalabs.com/products/hakrawler
Homepage: https://hakluke.com
Language: Go
License: GPL-3.0
License Family: copyleft
Topics: bugbounty, crawling, hacking, osint, pentesting, recon, reconnaissance
Last push: 2026-08-05T04:05:39+00:00

## Health v2 (maintenance only)
Score: 66/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 96, release rhythm 8, longevity 100
- inputs: {"age_days": 2453, "days_push": 28, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 5116, forks 536 (observed 2026-08-28T04:09:10.833431+00:00)

## What it is
Hakrawler is a fast command-line web crawler written in Go, built on the Gocolly library, that discovers URLs and JavaScript file locations from web applications. It reads URLs from stdin and fits naturally into Unix pipelines for reconnaissance workflows.

## Use cases
- crawl a website to discover endpoints and assets
- find javascript files on a target domain for bug bounty recon
- pipe subdomains from httpx into a crawler to enumerate urls
- discover hidden paths and forms during penetration testing
- gather urls from a list of domains quickly from the command line
- crawl a site through a proxy for traffic inspection

## When to choose
- you need fast, scriptable endpoint discovery in a recon pipeline
- you want a lightweight stdin/stdout tool that chains with httpx or subfinder
- you're doing bug bounty or pentest reconnaissance and need URL and JS file discovery

## When to avoid
- you need a full-featured crawler with rendering of JavaScript-heavy SPAs
- you want a GUI or interactive crawling experience
- you need polite, rate-limited crawling of sites you don't own without careful configuration

## Facets
- artifact type: cli-tool
- maturity: active
- function: web-scraping, cli, http-client
- domain: security, penetration-testing, osint, web-development
- platform: windows, cli, go
- tags: bug-bounty, recon, endpoint-discovery, javascript-discovery, stdin-pipeline, gocolly, command-line, linux, macos, docker

## Member repositories
- hakluke/hakrawler (main) score 66

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:09:10.833431+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T18:02:07.436909+00:00, confidence not recorded.
  - readme: https://github.com/hakluke/hakrawler (fetched 2026-08-28T04:09:10.833431+00:00, sha d77a92dc8ce1)
  - homepage: https://hakluke.com (fetched 2026-08-29T08:56:47.263656+00:00, sha 90fbf5cfa124)
- Data as of 2026-08-30T08:39:29.467469+00:00.