meta-llama/synthetic-data-kit

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1632 stars · 236 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A Python CLI tool from Meta for generating high-quality synthetic datasets to fine-tune LLMs. It follows a four-stage pipeline (ingest, create, curate, save-as) that converts documents into QA pairs, chain-of-thought reasoning traces, and summaries in fine-tuning-friendly formats.

Use cases

• generate synthetic QA pairs from pdfs for fine-tuning
• create chain-of-thought reasoning traces with an LLM
• convert documents into fine-tuning friendly chat formats
• curate synthetic datasets using LLM-as-a-judge
• prepare training data for Llama fine-tuning
• ingest pdfs, html, youtube transcripts, docx and ppt files
• export datasets to formats required by fine-tuning packages

When to choose

• you need structured training data in user/assistant format for LLM fine-tuning
• you want an end-to-end pipeline from raw documents to curated fine-tuning datasets
• you use vLLM or a local/external LLM endpoint and want modular dataset generation
• you want LLM-as-a-judge curation to filter low-quality examples

When to avoid

• you need general-purpose data labeling or annotation for non-LLM ML tasks
• you want a GUI or web interface rather than a CLI workflow
• you don't have access to an LLM backend for generation
• you need real-time or streaming data generation rather than batch pipelines

Facets

cli-tool · maturity active

data-generation llm-training etl cli nlp machine-learning large-language-models developer-tools python cli cross-platform synthetic-data fine-tuning qa-pairs chain-of-thought llama vllm data-curation lance-format data-engineering

Member repositories

For agents

markdown · JSON · MCP: product_card(name="meta-llama/synthetic-data-kit")