Eventual-Inc/Daft

Health v2 · maintenance only

Adoption not part of the score

5804 stars · 575 forks observed · 2026-10-16

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Daft is a high-performance distributed data engine with a Python dataframe API, implemented in Rust, designed for AI and multimodal workloads. It processes images, audio, video, embeddings, and structured data at scale, with built-in model operators and scaling from local to Ray or Kubernetes clusters.

Use cases

• process images and video at scale for ML training data
• generate embeddings for a vector database
• run LLM extraction over large datasets
• build multimodal AI ETL pipelines
• prepare training-ready datasets from S3 or Iceberg
• distributed dataframe processing without Spark's JVM
• run GPU inference alongside CPU data transforms

When to choose

• you need Pandas/Spark-like dataframe operations on multimodal data
• you want to mix CPU decoding and GPU inference in one pipeline
• you need to scale from laptop to Ray or Kubernetes clusters
• you work with Parquet, Iceberg, Delta Lake, or Hugging Face datasets

When to avoid

• you only need small in-memory analytics that Pandas handles fine
• your workloads are purely tabular with no AI or multimodal needs
• you require a JVM-based Spark ecosystem with its integrations

Facets

library · maturity active

etl data-science machine-learning rag llm-inference image-processing audio-processing video-processing streaming machine-learning artificial-intelligence big-data data-science python rust cloud cross-platform dataframe distributed-computing multimodal parquet iceberg ray embeddings gpu arrow ai-pipeline data-engineering docker kubernetes

Member repositories

For agents

markdown · JSON · MCP: product_card(name="Eventual-Inc/Daft")