elixir-crawly/crawly

Health v2 · maintenance only

Adoption not part of the score

1114 stars · 123 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Crawly is a high-level web crawling and scraping framework for Elixir, modeled after Scrapy, where developers define spiders that fetch pages and extract structured data. It ships with built-in support for concurrent requests, retries, request throttling, automatic link following, and pluggable item-processing pipelines.

Use cases

• scrape product prices and titles from e-commerce sites
• crawl a website and extract structured data into a database
• build a scalable web crawler in Elixir
• follow pagination links automatically while collecting items
• collect news articles or listings for data mining
• archive website content periodically
• extract data from HTML pages for downstream ETL pipelines

When to choose

• You are building a medium-to-large scale crawler in Elixir and want BEAM concurrency for parallel fetches
• You want a Scrapy-like workflow with spiders, item pipelines, and configurable middleware
• You need out-of-the-box features like retries, rate limiting, robots.txt handling, and automatic URL deduplication
• You want structured data extraction integrated with parsers like Floki

When to avoid

• You only need a one-off simple HTTP fetch, where a plain HTTP client like Req or HTTPoison is simpler
• You need to scrape heavily JavaScript-rendered pages without setting up browser rendering yourself
• Your project is not on the Elixir/Erlang platform
• You need a turnkey scraping service rather than a framework you write spider code in

Facets

framework · maturity active

web-scraping http-client parser etl crawlers web-development data-science elixir windows bsd cross-platform crawler spider data-extraction html-parsing scrapy-inspired erlang beam-concurrency data-engineering linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="elixir-crawly/crawly")