elixir-broadway/broadway

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

2677 stars · 179 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Broadway is an Elixir library for building concurrent, multi-stage data ingestion and processing pipelines on top of GenStage. It provides built-in back-pressure, batching, acknowledgements, rate-limiting, fault tolerance, and integrations with producers like Amazon SQS, Kafka, RabbitMQ, and Google Pub/Sub.

Use cases

• consume SQS messages concurrently in Elixir
• build a Kafka event processing pipeline
• batch process RabbitMQ messages and write to PostgreSQL
• ingest Google Pub/Sub events with back-pressure
• add rate limiting to a message processing pipeline
• process large volumes of data concurrently with fault tolerance

When to choose

• you need long-lived, concurrent ETL or event-processing pipelines in Elixir
• you want built-in back-pressure, batching, and automatic acknowledgements
• you consume from supported producers like SQS, Kafka, RabbitMQ, or Pub/Sub
• you need ordering, partitioning, and graceful shutdown out of the box

When to avoid

• you need one-off batch scripts rather than long-lived pipelines
• your stack is not Elixir/Erlang
• you need a full stream processing engine with windowing like Flink or Spark Streaming

Facets

library · maturity stable

streaming message-queue concurrency rate-limiting monitoring etl backend microservices elixir jvm genstage data-ingestion data-processing backpressure batching sqs kafka rabbitmq pubsub fault-tolerance data-engineering messaging automation

Member repositories

For agents

markdown · JSON · MCP: product_card(name="elixir-broadway/broadway")