# datachain-ai/datachain

The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure

Repository: https://github.com/datachain-ai/datachain
Canonical: https://ross.abutalabs.com/products/datachain
Homepage: https://docs.datachain.ai
Language: Python
License: Apache-2.0
License Family: permissive
Topics: claude-code, codex, data-processing, harness-engineering, mlops, unstructured-data, data-context-layer, ai-agents, knowledge-base, multimodal, pydantic
Last push: 2026-08-26T14:34:25+00:00

## Health v2 (maintenance only)
Score: 86/100 (v2, computed 2026-09-03T02:39:23.370411+00:00)
- activity 99, release rhythm 87, longevity 57
- inputs: {"age_days": 799, "days_push": 7, "days_rel": 9, "gap_med": 1, "n_releases_24m": 250}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 2811, forks 156 (observed 2026-08-28T04:07:23.670050+00:00)

## What it is
DataChain is a Python library that turns unstructured files in S3, GCS, and Azure into typed, versioned datasets queryable at warehouse speed, with a distributed compute engine for parallel processing over files. It also provides an optional Knowledge Base and Agent Harness (skill + MCP server) that plug into Claude Code, Cursor, Codex, and other coding agents so they can understand and reuse your data.

## Use cases
- process and curate unstructured data in S3, GCS, or Azure at scale
- build versioned, typed datasets from files in cloud object storage
- run batch inference with ML models over images or text in cloud storage
- give coding agents like Claude Code persistent knowledge of my datasets
- find images similar to a reference image in a large cloud bucket
- filter and join files with JSON metadata annotations
- avoid recomputing embeddings and filters across agent sessions

## When to choose
- you work with large volumes of unstructured files (images, text, video) in S3, GCS, or Azure and need typed, versioned, queryable datasets
- you want distributed, checkpointed Python processing over cloud files without moving bytes out of storage
- you use AI coding agents and want them to reuse prior data-processing results instead of starting from raw files each session
- you need warehouse-speed filtering, joining, or similarity search over hundreds of millions of file records

## When to avoid
- your data is already structured and lives in a relational database or data warehouse with existing tooling
- you need a full visual data-pipeline orchestration platform rather than a Python library
- your datasets are small enough that pandas or plain scripts suffice
- you cannot use cloud object storage and need only local single-machine processing

## Facets
- artifact type: library
- maturity: active
- function: etl, data-science, machine-learning, rag, mcp, search-engine, file-system, cli
- domain: machine-learning, artificial-intelligence, developer-tools, cloud-computing
- platform: python, cli, cloud, cross-platform
- tags: unstructured-data, data-versioning, object-storage, pydantic, mlops, multimodal, agent-harness, knowledge-base, data-curation, s3, gcs, azure, data-engineering, ai-agents, docker

## Member repositories
- datachain-ai/datachain (main) score 86

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:07:23.670050+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T08:13:32.166940+00:00, confidence not recorded.
  - readme: https://github.com/datachain-ai/datachain (fetched 2026-08-28T04:07:23.670050+00:00, sha fd6a056021c7)
  - homepage: https://docs.datachain.ai (fetched 2026-08-29T09:54:07.625978+00:00, sha 99e362052d80)
  - site_page: https://docs.datachain.ai/getting-started/python (fetched 2026-08-29T09:54:07.635083+00:00, sha cbbc871baacb)
  - site_page: https://docs.datachain.ai/getting-started/agents (fetched 2026-08-29T09:54:07.637223+00:00, sha 28f67aeaa66c)
  - registry_pypi: https://pypi.org/pypi/datachain/json (fetched 2026-08-29T09:54:07.638902+00:00, sha a74a64729a96)
- Data as of 2026-08-30T08:39:29.467469+00:00.