# baidu/bigflow

Baidu Bigflow is an interface that allows for writing distributed computing programs and provides lots of simple, flexible, powerful APIs. Using Bigflow, you can easily handle data of any scale. Bigflow processes 4P+ data inside Baidu and runs about 10k jobs every day.

Repository: https://github.com/baidu/bigflow
Canonical: https://ross.abutalabs.com/products/bigflow
Homepage: http://baidu.github.io/bigflow
Language: C++
License: Apache-2.0
License Family: permissive
Last push: 2025-12-11T20:38:47+00:00

## Health v2 (maintenance only)
Score: 48/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 56, release rhythm 8, longevity 100
- inputs: {"age_days": 3214, "days_push": 265, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1131, forks 159 (observed 2026-08-28T04:03:42.527140+00:00)

## What it is
Baidu Bigflow is a distributed computing framework offering simple, flexible Python APIs for writing data processing programs that can run on different execution engines. It is inspired by FlumeJava, Cloud Dataflow, and Spark, with an open-source release targeting Spark as the backend.

## Use cases
- write distributed data processing jobs in python
- run batch etl pipelines on spark
- replace pyspark with a faster python-first framework
- process large-scale datasets without managing partitioners
- port single-machine python logic to distributed execution

## When to choose
- you want a python-first distributed computing api with better performance than pyspark
- you want code portable across execution engines
- you prefer spark-like apis with simpler semantics like nested distributed datasets

## When to avoid
- you need streaming engines other than spark, since the open-source version only supports bigflow on spark
- you need a large community and ecosystem, as bigflow is baidu-centric with limited external adoption
- you work primarily in scala or java rather than python

## Facets
- artifact type: framework
- maturity: active
- function: etl, streaming, data-science, workflow-automation
- domain: big-data, microservices, data-science
- platform: python, cpp
- tags: distributed-computing, flumejava, batch-processing, pyspark-alternative, data-pipeline, data-engineering, spark, linux, docker

## Member repositories
- baidu/bigflow (main) score 48

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:03:42.527140+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T06:38:01.092682+00:00, confidence not recorded.
  - readme: https://github.com/baidu/bigflow (fetched 2026-08-28T04:03:42.527140+00:00, sha cc49c8940282)
  - homepage: http://baidu.github.io/bigflow (fetched 2026-08-29T12:42:24.389024+00:00, sha 82c2723a3b55)
- Data as of 2026-08-30T08:39:29.467469+00:00.