PolyAI-LDN/conversational-datasets resource

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1401 stars · 177 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A collection of tools and scripts from PolyAI for generating large, reproducible datasets for conversational response selection, including Reddit (3.7B comments), OpenSubtitles (400M+ lines), and Amazon QA (3.6M pairs). Rather than shipping raw data, it provides dataflow scripts with deterministic train/test splits so researchers can regenerate and customize the preprocessing.

Use cases

• pre-train conversational response selection models on large dialogue corpora
• generate reproducible train/test splits for NLP research papers
• build a chatbot training dataset from Reddit comment threads
• create a dialogue dataset from movie and TV subtitles
• benchmark retrieval-based conversation models
• customize filtering and preprocessing of large conversational corpora

When to choose

• you need massive-scale conversational data for pre-training or fine-tuning response selection models
• you want reproducible, deterministic dataset splits for research evaluation
• you want control over the preprocessing and filtering pipeline

When to avoid

• you need ready-to-download processed data rather than scripts that regenerate it
• you need small, curated dialogue datasets for domain-specific chatbots
• you lack the storage and compute to process billions of raw records
• you need datasets for generative dialogue rather than response selection

Facets

dataset · maturity maintenance

machine-learning nlp data-generation etl machine-learning chatbots artificial-intelligence python cross-platform conversational-ai response-selection reddit-dataset opensubtitles amazon-qa tensorflow reproducible-benchmarks natural-language-processing linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="PolyAI-LDN/conversational-datasets")