hakluke/hakrawler

Health v2 · maintenance only

Adoption not part of the score

5116 stars · 536 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Hakrawler is a fast command-line web crawler written in Go, built on the Gocolly library, that discovers URLs and JavaScript file locations from web applications. It reads URLs from stdin and fits naturally into Unix pipelines for reconnaissance workflows.

Use cases

• crawl a website to discover endpoints and assets
• find javascript files on a target domain for bug bounty recon
• pipe subdomains from httpx into a crawler to enumerate urls
• discover hidden paths and forms during penetration testing
• gather urls from a list of domains quickly from the command line
• crawl a site through a proxy for traffic inspection

When to choose

• you need fast, scriptable endpoint discovery in a recon pipeline
• you want a lightweight stdin/stdout tool that chains with httpx or subfinder
• you're doing bug bounty or pentest reconnaissance and need URL and JS file discovery

When to avoid

• you need a full-featured crawler with rendering of JavaScript-heavy SPAs
• you want a GUI or interactive crawling experience
• you need polite, rate-limited crawling of sites you don't own without careful configuration

Facets

cli-tool · maturity active

web-scraping cli http-client security penetration-testing osint web-development windows cli go bug-bounty recon endpoint-discovery javascript-discovery stdin-pipeline gocolly command-line linux macos docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="hakluke/hakrawler")