OpenDCAI/DataFlow

Health v2 · maintenance only

Adoption not part of the score

7764 stars · 1047 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

DataFlow is a Python framework for preparing, cleaning, and synthesizing data for LLM training and RAG using composable LLM-based operators and pipelines. It ships with a CLI, a Gradio interface, and backends for vLLM and SGLang inference.

Use cases

• clean and filter training data for LLM fine-tuning
• synthesize instruction data with LLM operators
• build data preparation pipelines for RAG
• deduplicate and quality-score large text corpora
• run data processing pipelines backed by vLLM or SGLang
• prepare question-answer datasets from raw documents

When to choose

• you need LLM-powered data cleaning or synthesis pipelines in Python
• you want composable operators for building training or RAG datasets
• you want a CLI plus GUI for quick data processing

When to avoid

• you need simple non-LLM ETL without model inference costs
• you need a fully managed cloud data platform
• your data is small enough for plain pandas scripts

Facets

framework · maturity active

etl data-science llm-inference rag workflow-automation machine-learning large-language-models data-science developer-tools python cross-platform cli self-hosted data-cleaning data-synthesis llm-operators data-pipelines vllm sglang gradio data-agent data-engineering natural-language-processing docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="OpenDCAI/DataFlow")