# chiphuyen/lazynlp

Library to scrape and clean web pages to create massive datasets.

Repository: https://github.com/chiphuyen/lazynlp
Canonical: https://ross.abutalabs.com/products/lazynlp
Language: Python
License Family: other
Topics: artificial-intelligence, natural-language-processing, nlp, text-mining, language-model, python, open, data-science
Last push: 2020-11-11T12:16:30+00:00

## Health v2 (maintenance only)
Score: 23/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 0, release rhythm 8, longevity 100
- inputs: {"age_days": 2744, "days_push": 2121, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_license
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 2284, forks 324 (observed 2026-08-28T04:06:34.000095+00:00)

## What it is
A Python library for crawling, cleaning, and deduplicating web pages to build massive monolingual text datasets, suitable for training language models. It provides helpers for gathering URL lists from sources like Reddit, Gutenberg, and Wikipedia and processing them into large corpora.

## Use cases
- scrape web pages to build a large text corpus
- create a dataset larger than GPT-2's training data
- deduplicate URLs before crawling
- clean and normalize scraped web text
- collect text from Reddit, Gutenberg, and Wikipedia dumps
- build monolingual datasets for language model training

## When to choose
- you need to build a large monolingual text corpus from web pages
- you want to crawl and deduplicate URLs at scale in Python
- you're preparing pretraining data for language models

## When to avoid
- you need a maintained tool with active support
- you want a no-code scraping solution
- you need JavaScript-rendered page scraping
- you require a license for commercial use clarity

## Facets
- artifact type: library
- maturity: abandoned
- function: web-scraping, nlp, etl, data-generation
- domain: artificial-intelligence, crawlers
- platform: python, cli
- tags: web-crawling, text-cleaning, deduplication, language-model-datasets, corpus-building, natural-language-processing, data-engineering

## Member repositories
- chiphuyen/lazynlp (main) score 23

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:06:34.000095+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T02:41:05.888113+00:00, confidence not recorded.
  - readme: https://github.com/chiphuyen/lazynlp (fetched 2026-08-28T04:06:34.000095+00:00, sha 771b96ae9a89)
- Data as of 2026-08-30T08:39:29.467469+00:00.