sjdirect/abot

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

2310 stars · 546 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Abot is an open source C# web crawler framework built for speed and flexibility, handling multithreading, HTTP requests, scheduling, and link parsing. It exposes a pluggable architecture where users register for events or swap in custom implementations to control the crawl process.

Use cases

• crawl a website and extract page data in C#
• build a custom web spider with pluggable crawl rules
• scrape links and page content from many pages quickly
• write a multithreaded crawler without managing HTTP plumbing
• parse links from HTML pages in .NET

When to choose

• you need a fast, lightweight .NET crawler with no external dependencies
• you want full control over crawl behavior via pluggable interfaces
• you need cross-platform support via .NET Standard 2.0

When to avoid

• you need JavaScript rendering or anti-blocking features out of the box (use AbotX or other tools)
• you want a turnkey scraping solution with no coding
• your project is not on the .NET platform

Facets

library · maturity active

web-scraping parser concurrency http-client crawlers web-development developer-tools cross-platform dotnet windows c-sharp web-crawler spider pluggable-architecture netstandard nuget linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="sjdirect/abot")