apache/beam

Health v2 · maintenance only

Adoption not part of the score

8650 stars · 4630 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Apache Beam is an open-source unified programming model and SDK set (Java, Python, Go, SQL, TypeScript) for defining batch and streaming data-parallel processing pipelines. Pipelines are portable across execution runners including Apache Flink, Apache Spark, Google Cloud Dataflow, Samza, and Hazelcast Jet.

Use cases

• build etl pipelines that run on multiple engines
• process streaming data in near real time at scale
• run batch and streaming with one codebase
• run ml model inference over large datasets
• migrate pipelines between spark flink and dataflow
• process trillions of events daily in production

When to choose

• you need one pipeline codebase for both batch and streaming
• you want portability across Flink, Spark, Dataflow, and other runners
• you need large-scale distributed data processing with strong abstractions
• you want to run ML inference as part of big data pipelines

When to avoid

• you have small datasets that fit on one machine
• you want a simple scheduler rather than a full programming model
• your team cannot invest in learning the Beam model
• you are locked into a single engine's native APIs and don't need portability

Facets

framework · maturity stable

etl streaming data-science machine-learning big-data microservices data-science machine-learning python jvm go cross-platform cloud batch-processing stream-processing data-pipelines apache-flink apache-spark google-cloud-dataflow runinference multi-language-pipelines unified-programming-model sql data-engineering docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="apache/beam")