chiphuyen/lazynlp

Health v2 · maintenance only

Flags: no_license

Adoption not part of the score

2284 stars · 324 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A Python library for crawling, cleaning, and deduplicating web pages to build massive monolingual text datasets, suitable for training language models. It provides helpers for gathering URL lists from sources like Reddit, Gutenberg, and Wikipedia and processing them into large corpora.

Use cases

• scrape web pages to build a large text corpus
• create a dataset larger than GPT-2's training data
• deduplicate URLs before crawling
• clean and normalize scraped web text
• collect text from Reddit, Gutenberg, and Wikipedia dumps
• build monolingual datasets for language model training

When to choose

• you need to build a large monolingual text corpus from web pages
• you want to crawl and deduplicate URLs at scale in Python
• you're preparing pretraining data for language models

When to avoid

• you need a maintained tool with active support
• you want a no-code scraping solution
• you need JavaScript-rendered page scraping
• you require a license for commercial use clarity

Facets

library · maturity abandoned

web-scraping nlp etl data-generation artificial-intelligence crawlers python cli web-crawling text-cleaning deduplication language-model-datasets corpus-building natural-language-processing data-engineering

Member repositories

For agents

markdown · JSON · MCP: product_card(name="chiphuyen/lazynlp")