datachain-ai/datachain

Health v2 · maintenance only

Adoption not part of the score

2811 stars · 156 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

DataChain is a Python library that turns unstructured files in S3, GCS, and Azure into typed, versioned datasets queryable at warehouse speed, with a distributed compute engine for parallel processing over files. It also provides an optional Knowledge Base and Agent Harness (skill + MCP server) that plug into Claude Code, Cursor, Codex, and other coding agents so they can understand and reuse your data.

Use cases

• process and curate unstructured data in S3, GCS, or Azure at scale
• build versioned, typed datasets from files in cloud object storage
• run batch inference with ML models over images or text in cloud storage
• give coding agents like Claude Code persistent knowledge of my datasets
• find images similar to a reference image in a large cloud bucket
• filter and join files with JSON metadata annotations
• avoid recomputing embeddings and filters across agent sessions

When to choose

• you work with large volumes of unstructured files (images, text, video) in S3, GCS, or Azure and need typed, versioned, queryable datasets
• you want distributed, checkpointed Python processing over cloud files without moving bytes out of storage
• you use AI coding agents and want them to reuse prior data-processing results instead of starting from raw files each session
• you need warehouse-speed filtering, joining, or similarity search over hundreds of millions of file records

When to avoid

• your data is already structured and lives in a relational database or data warehouse with existing tooling
• you need a full visual data-pipeline orchestration platform rather than a Python library
• your datasets are small enough that pandas or plain scripts suffice
• you cannot use cloud object storage and need only local single-machine processing

Facets

library · maturity active

etl data-science machine-learning rag mcp search-engine file-system cli machine-learning artificial-intelligence developer-tools cloud-computing python cli cloud cross-platform unstructured-data data-versioning object-storage pydantic mlops multimodal agent-harness knowledge-base data-curation s3 gcs azure data-engineering ai-agents docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="datachain-ai/datachain")