WikiExtractor/wikiextractor
Some data may be outdated — see provenance labels.
A tool for extracting plain text from Wikipedia dumps observed · 2026-10-03
Health v2 · maintenance only
95/100
- Activity 97
- Release rhythm 90
- Longevity 100
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-10-21. Adoption (stars, forks) is never an input.
- gap_med: 18
- age_days: 4230
- days_rel: 71
- days_push: 22
- n_releases_24m: 2
Adoption not part of the score
4005 stars · 998 forks observed · 2026-10-03
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded
WikiExtractor is a Python command-line tool that extracts and cleans plain text from Wikipedia database backup dumps (XML). It supports template expansion, multiprocessing, and JSON or custom document output formats.
Use cases
- extract plain text from a Wikipedia dump
- build a text corpus for NLP training from Wikipedia
- convert Wikipedia XML dumps to JSON documents
- preprocess Wikipedia articles for language model training
- extract a single page from a Wikipedia dump
- strip MediaWiki markup and templates from articles
When to choose
- you need clean plain text from full Wikipedia dumps
- you want a dependency-free Python tool
- you need parallel extraction of large dumps
- you want template expansion or JSON output
When to avoid
- you need live Wikipedia API access rather than offline dumps
- you're on Windows, where StringIO issues cause problems
- you need other MediaWiki-based wikis with complex custom templates
- you need structured data like wikidata triples
Facets
cli-tool · maturity stable
parser nlp etl cli developer-tools python cli cross-platform wikipedia text-extraction wiki-dump mediawiki corpus-preparation natural-language-processing data-engineering linux macos
2 sources
- readme: https://github.com/WikiExtractor/wikiextractor · fetched 2026-10-03 · 52562b6d4232
- homepage: http://wikiextractor.github.io/wikiextractor/ · fetched 2026-08-29 · b8d76997114d
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| WikiExtractor/wikiextractor | main | 95 |
For agents
markdown · JSON · MCP: product_card(name="WikiExtractor/wikiextractor")
Data as of 2026-10-19T08:05:51.212177+00:00 · Report a problem