arkflow-rs/arkflow

Health v2 · maintenance only

Adoption not part of the score

1302 stars · 46 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

ArkFlow is a high-performance stream processing engine written in Rust on top of Tokio, connecting configurable inputs (Kafka, MQTT, HTTP, files, SQL databases), processors (SQL via DataFusion, JSON, Protobuf, VRL, Python UDFs), and outputs into YAML-defined pipelines. It integrates AI capabilities such as loading machine learning models for streaming inference and anomaly detection, and offers an optional control plane for operating fleets of nodes.

Use cases

• process kafka streams with sql queries in real time
• run machine learning inference on streaming data
• build an iot mqtt data pipeline
• transform and route events between systems
• process debezium cdc change streams
• detect anomalies in real-time sensor data

When to choose

• you need a lightweight, high-performance alternative to Flink or Kafka Streams for stateless pipelines
• you want SQL, VRL, or Python UDF processing over streaming data in a single YAML-configured binary
• you need streaming inference or anomaly detection with ML models embedded in the pipeline

When to avoid

• you need full stateful stream processing with windowing, joins, and exactly-once state management today
• you require a mature, battle-tested engine with a large community and ecosystem
• your pipelines depend on connectors or processors not yet supported by its modular component set

Facets

application · maturity active

streaming etl machine-learning serialization message-queue big-data machine-learning iot windows rust self-hosted stream-processing datafusion kafka mqtt cdc vrl arrow tokio real-time-pipelines ai-inference sql data-engineering real-time linux macos docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="arkflow-rs/arkflow")