scrapinghub/frontera

Health v2 · maintenance only

Adoption not part of the score

1332 stars · 212 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Frontera is a Python web crawling framework that implements a scalable crawl frontier, storing and prioritizing links extracted by crawlers to decide which pages to visit next. It supports both single-process and distributed operation with pluggable backends (SQLAlchemy, Redis, HBase) and message buses (Kafka, ZeroMQ).

Use cases

• build a large-scale distributed web crawler
• prioritize which URLs a crawler should fetch next
• run a breadth-first or depth-first crawl strategy at scale
• integrate a crawl frontier with Scrapy spiders
• distribute crawling work across multiple workers via Kafka
• implement a site discovery crawler respecting robots.txt and sitemaps

When to choose

• you need a battle-tested, scalable frontier for a large online crawler
• you want pluggable backends and message buses for distributed crawling
• you already use Scrapy and need frontier logic
• you need to crawl tens of millions of documents daily

When to avoid

• you only need a simple one-off scraper without URL prioritization
• you want a turnkey crawler rather than a frontier framework to build on
• you need active feature development - the project is largely in maintenance mode

Facets

framework · maturity maintenance

web-scraping message-queue caching database crawlers web-development big-data python self-hosted crawl-frontier distributed-crawling scrapy kafka hbase redis url-prioritization search linux docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="scrapinghub/frontera")