TimelyDataflow/timely-dataflow

Health v2 · maintenance only

Adoption not part of the score

3642 stars · 293 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

A modular Rust implementation of the timely dataflow computational model from the Naiad paper, providing a low-latency cyclic data-parallel compute engine. The same program scales from a single thread on a laptop to distributed execution across a cluster.

Use cases

• build low-latency streaming dataflow pipelines in Rust
• scale a data-parallel program from one thread to a cluster
• implement incremental and cyclic graph computations
• prototype distributed stream processing algorithms
• process large datasets with high-throughput parallel operators

When to choose

• you need expressive, low-latency cyclic dataflow computation in Rust
• you want one program that scales from laptop to cluster
• you are building stream processing systems or research compute engines

When to avoid

• you need a turnkey streaming system with connectors and SQL rather than a compute framework
• your team is not comfortable with Rust and dataflow programming
• you only need simple batch ETL with existing tooling

Facets

library · maturity active

streaming concurrency data-science etl microservices big-data rust windows cross-platform dataflow naiad data-parallel low-latency distributed-compute data-engineering real-time linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="TimelyDataflow/timely-dataflow")