# OpenDCAI/DataFlow

Easy Data Preparation with latest LLMs-based Operators and Pipelines.

Repository: https://github.com/OpenDCAI/DataFlow
Canonical: https://ross.abutalabs.com/products/dataflow
Homepage: https://OpenDCAI.github.io/DataFlow-Doc/
Language: Python
License: Apache-2.0
License Family: permissive
Topics: data, data-cleaning, data-pipelines, data-processing, data-science, data-synthesis, llms, operators, data-agent, sglang-bankend, vllm-backend, gradio-interface, quick-data-processing
Last push: 2026-08-18T12:18:15+00:00

## Health v2 (maintenance only)
Score: 80/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 98, release rhythm 76, longevity 49
- inputs: {"age_days": 689, "days_push": 15, "days_rel": 160, "gap_med": 7, "n_releases_24m": 12}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 7764, forks 1047 (observed 2026-08-28T04:10:04.831926+00:00)

## What it is
DataFlow is a Python framework for preparing, cleaning, and synthesizing data for LLM training and RAG using composable LLM-based operators and pipelines. It ships with a CLI, a Gradio interface, and backends for vLLM and SGLang inference.

## Use cases
- clean and filter training data for LLM fine-tuning
- synthesize instruction data with LLM operators
- build data preparation pipelines for RAG
- deduplicate and quality-score large text corpora
- run data processing pipelines backed by vLLM or SGLang
- prepare question-answer datasets from raw documents

## When to choose
- you need LLM-powered data cleaning or synthesis pipelines in Python
- you want composable operators for building training or RAG datasets
- you want a CLI plus GUI for quick data processing

## When to avoid
- you need simple non-LLM ETL without model inference costs
- you need a fully managed cloud data platform
- your data is small enough for plain pandas scripts

## Facets
- artifact type: framework
- maturity: active
- function: etl, data-science, llm-inference, rag, workflow-automation, machine-learning
- domain: large-language-models, data-science, developer-tools
- platform: python, cross-platform, cli, self-hosted
- tags: data-cleaning, data-synthesis, llm-operators, data-pipelines, vllm, sglang, gradio, data-agent, data-engineering, natural-language-processing, docker

## Member repositories
- OpenDCAI/DataFlow (main) score 80

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:10:04.831926+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T17:35:07.768727+00:00, confidence not recorded.
  - readme: https://github.com/OpenDCAI/DataFlow (fetched 2026-08-28T04:10:04.831926+00:00, sha f4be5e6b3c35)
  - homepage: https://OpenDCAI.github.io/DataFlow-Doc/ (fetched 2026-08-29T08:31:03.669460+00:00, sha 1260f20fe66e)
- Data as of 2026-08-30T08:39:29.467469+00:00.