numaproj/numaflow

Health v2 · maintenance only

Adoption not part of the score

2825 stars · 171 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Numaflow is a Kubernetes-native, serverless platform for running massively parallel data and streaming processing jobs. It lets developers build event-driven pipelines with sources, sinks, and user-defined functions in any language, with auto-scaling and exactly-once semantics.

Use cases

• process kafka events in real time on kubernetes
• run streaming data pipelines with auto-scaling
• real-time analytics on event streams
• run ml inference on streaming data
• build event-driven applications with exactly-once guarantees
• run map-reduce style jobs on kubernetes
• write pipeline processing steps in python or go

When to choose

• you already run workloads on Kubernetes and want declarative streaming pipelines
• you need independent auto-scaling per pipeline stage with back-pressure
• you want language-agnostic user-defined functions (Python, Go, Java, Rust)
• you need exactly-once or at-least-once processing guarantees for streaming data

When to avoid

• you have no Kubernetes cluster and only need simple batch ETL
• you need strict message ordering guarantees
• you want a fully managed cloud streaming service rather than self-hosted infrastructure
• your project cannot tolerate occasional breaking changes between minor releases

Facets

framework · maturity active

streaming etl workflow-automation microservices deployment microservices big-data cloud-computing cloud self-hosted go rust python jvm stream-processing event-driven serverless map-reduce auto-scaling exactly-once kubernetes-operator data-pipeline data-engineering real-time containers kubernetes docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="numaproj/numaflow")