# PolyAI-LDN/conversational-datasets

Large datasets for conversational AI

Repository: https://github.com/PolyAI-LDN/conversational-datasets
Canonical: https://ross.abutalabs.com/products/conversational-datasets
Language: Python
License: Apache-2.0
License Family: permissive
Topics: conversational-ai, datasets, machine-learning
Last push: 2019-11-16T03:50:49+00:00

## Health v2 (maintenance only)
Score: 32/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 0, release rhythm 35, longevity 100
- inputs: {"age_days": 2736, "days_push": 2482, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1401, forks 177 (observed 2026-08-28T04:04:37.237610+00:00)

## What it is
A collection of tools and scripts from PolyAI for generating large, reproducible datasets for conversational response selection, including Reddit (3.7B comments), OpenSubtitles (400M+ lines), and Amazon QA (3.6M pairs). Rather than shipping raw data, it provides dataflow scripts with deterministic train/test splits so researchers can regenerate and customize the preprocessing.

## Use cases
- pre-train conversational response selection models on large dialogue corpora
- generate reproducible train/test splits for NLP research papers
- build a chatbot training dataset from Reddit comment threads
- create a dialogue dataset from movie and TV subtitles
- benchmark retrieval-based conversation models
- customize filtering and preprocessing of large conversational corpora

## When to choose
- you need massive-scale conversational data for pre-training or fine-tuning response selection models
- you want reproducible, deterministic dataset splits for research evaluation
- you want control over the preprocessing and filtering pipeline

## When to avoid
- you need ready-to-download processed data rather than scripts that regenerate it
- you need small, curated dialogue datasets for domain-specific chatbots
- you lack the storage and compute to process billions of raw records
- you need datasets for generative dialogue rather than response selection

## Facets
- artifact type: dataset
- maturity: maintenance
- function: machine-learning, nlp, data-generation, etl
- domain: machine-learning, chatbots, artificial-intelligence
- platform: python, cross-platform
- tags: conversational-ai, response-selection, reddit-dataset, opensubtitles, amazon-qa, tensorflow, reproducible-benchmarks, natural-language-processing, linux

## Member repositories
- PolyAI-LDN/conversational-datasets (main) score 32

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:04:37.237610+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T04:39:00.946278+00:00, confidence not recorded.
  - readme: https://github.com/PolyAI-LDN/conversational-datasets (fetched 2026-08-28T04:04:37.237610+00:00, sha b9c5ba6ec9e7)
- Data as of 2026-08-30T08:39:29.467469+00:00.