databricks/lilac

Health v2 · maintenance only

Flags: archived

Adoption not part of the score

1072 stars · 105 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Lilac is an open-source tool for exploring, curating, and quality-controlling datasets used for training, fine-tuning, and monitoring LLMs. It provides both a web UI and Python API, running on-device with open-source LLMs for search, filtering, clustering, and annotation of unstructured data.

Use cases

• curate fine-tuning datasets for LLMs
• find and remove PII from training data
• deduplicate large text datasets before training
• explore and search millions of documents with semantic search
• cluster and label unstructured text data with LLMs
• monitor dataset quality changes over time
• visualize pre-training data distributions

When to choose

• you need to clean and curate large unstructured text datasets for LLM training or fine-tuning
• you want an interactive UI plus Python API for dataset exploration
• you need on-device processing with open-source LLMs for data quality tasks like PII removal and deduplication

When to avoid

• you need a fully managed enterprise data platform with support guarantees
• your data is structured tabular data rather than unstructured text
• you require active development and frequent updates, as the latest release is from March 2024

Facets

library · maturity maintenance

data-science etl search-engine nlp machine-learning data-visualization artificial-intelligence large-language-models data-science machine-learning python self-hosted cross-platform dataset-curation llm-data-quality data-exploration fine-tuning-data pii-detection semantic-search clustering web-ui data-engineering web-server

Member repositories

For agents

markdown · JSON · MCP: product_card(name="databricks/lilac")