# EleutherAI/the-pile

Repository: https://github.com/EleutherAI/the-pile
Canonical: https://ross.abutalabs.com/products/the-pile
Language: Python
License: MIT
License Family: permissive
Last push: 2023-04-27T21:00:48+00:00

## Health v2 (maintenance only)
Score: 32/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 0, release rhythm 35, longevity 100
- inputs: {"age_days": 2198, "days_push": 1224, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1673, forks 150 (observed 2026-08-28T04:05:20.322563+00:00)

## What it is
The Pile is a large, diverse, open-source language modeling dataset composed of many smaller text sources combined together. This repository contains the code for replicating the Pile or building variants of it, rather than the dataset itself.

## Use cases
- build a large text corpus for training language models
- replicate the Pile dataset from its component sources
- create a custom variant of the Pile with different data mixtures
- download diverse text data for pretraining research
- study the composition and weighting of an LLM pretraining dataset

## When to choose
- you need to reproduce or modify the Pile's data pipeline
- you want to understand how a large pretraining corpus is assembled from many sources
- you are doing research on dataset composition for language modeling

## When to avoid
- you just want to download and use the Pile dataset itself (get it from the official download link instead)
- you need a maintained, actively updated corpus pipeline
- you need a small curated dataset rather than hundreds of gigabytes of text

## Facets
- artifact type: dataset
- maturity: maintenance
- function: etl, data-generation, machine-learning, llm-training
- domain: large-language-models, machine-learning
- platform: python
- tags: language-modeling, text-corpus, dataset-replication, eleutherai, open-source-data, natural-language-processing, data-engineering, linux, macos

## Member repositories
- EleutherAI/the-pile (main) score 32

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:05:20.322563+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T03:42:34.343493+00:00, confidence not recorded.
  - readme: https://github.com/EleutherAI/the-pile (fetched 2026-08-28T04:05:20.322563+00:00, sha c278c93424d7)
- Data as of 2026-08-30T08:39:29.467469+00:00.