Pathway

Health v2 · maintenance only

Flags: no_license

Adoption not part of the score

62383 stars · 1679 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Pathway is a Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG applications. It provides a unified Python API that handles both batch and streaming data in development and production, backed by a Rust engine.

Use cases

• build real-time data pipelines from kafka streams
• process streaming data with python
• build a rag pipeline over live documents
• run llm pipelines on continuously updating data
• perform real-time analytics on iot sensor data
• unify batch and streaming etl in one framework
• index and query changing data for ai applications

When to choose

• you need one codebase for both batch and streaming data
• you are building RAG or LLM pipelines over continuously updating sources
• you want a Python API with a high-performance Rust runtime
• you need real-time analytics on kafka or iot streams

When to avoid

• you need a permissively licensed library for commercial redistribution (BSL license)
• you only need simple one-off batch transformations where pandas suffices
• you require a mature SQL-first streaming system like Flink

Facets

framework · maturity active

etl streaming rag machine-learning data-science llm-inference big-data large-language-models analytics python self-hosted cross-platform stream-processing real-time-analytics llm-pipelines kafka rust-engine bsl-license data-engineering retrieval-augmented-generation real-time docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="pathwaycom/pathway")