baidu/bigflow
Baidu Bigflow is an interface that allows for writing distributed computing programs and provides lots of simple, flexible, powerful APIs. Using Bigflow, you can easily handle data of any scale. Bigflow processes 4P+ data inside Baidu and runs about 10k jobs every day. observed · 2026-08-28
Health v2 · maintenance only
48/100
- Activity 56
- Release rhythm 8
- Longevity 100
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-03. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 3214
- days_rel: n/a
- days_push: 265
- n_releases_24m: 0
Adoption not part of the score
1131 stars · 159 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded
Baidu Bigflow is a distributed computing framework offering simple, flexible Python APIs for writing data processing programs that can run on different execution engines. It is inspired by FlumeJava, Cloud Dataflow, and Spark, with an open-source release targeting Spark as the backend.
Use cases
- write distributed data processing jobs in python
- run batch etl pipelines on spark
- replace pyspark with a faster python-first framework
- process large-scale datasets without managing partitioners
- port single-machine python logic to distributed execution
When to choose
- you want a python-first distributed computing api with better performance than pyspark
- you want code portable across execution engines
- you prefer spark-like apis with simpler semantics like nested distributed datasets
When to avoid
- you need streaming engines other than spark, since the open-source version only supports bigflow on spark
- you need a large community and ecosystem, as bigflow is baidu-centric with limited external adoption
- you work primarily in scala or java rather than python
Facets
framework · maturity active
etl streaming data-science workflow-automation big-data microservices data-science python cpp distributed-computing flumejava batch-processing pyspark-alternative data-pipeline data-engineering spark linux docker
2 sources
- readme: https://github.com/baidu/bigflow · fetched 2026-08-28 · cc49c8940282
- homepage: http://baidu.github.io/bigflow · fetched 2026-08-29 · 82c2723a3b55
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| baidu/bigflow | main | 48 |
For agents
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem