jfilter/clean-text

Health v2 · maintenance only

Flags: no_license

Adoption not part of the score

1027 stars · 83 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A Python package for cleaning and normalizing messy text, especially user-generated content from the web and social media. It fixes unicode errors, transliterates to ASCII, and replaces URLs, emails, phone numbers, and other noise with special tokens.

Use cases

• clean scraped web text before nlp processing
• normalize user-generated content from social media
• fix broken unicode characters in text
• replace urls and emails with tokens in text preprocessing
• transliterate accented characters to ascii
• preprocess text for machine learning models

When to choose

• you need to preprocess dirty scraped or user-generated text
• you want a simple pip-installable text cleaning pipeline
• you need unicode fixing and ascii transliteration in one package

When to avoid

• you need heavy-duty nlp pipelines with tokenization and parsing
• you require permissively-licensed dependencies only and cannot accept GPL unidecode
• you need language-specific linguistic cleaning rather than rule-based normalization

Facets

library · maturity active

nlp parser serialization text-editors crawlers machine-learning python cross-platform text-cleaning text-normalization text-preprocessing user-generated-content unicode regex natural-language-processing

Member repositories

For agents

markdown · JSON · MCP: product_card(name="jfilter/clean-text")