# adbar/trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

Repository: https://github.com/adbar/trafilatura
Canonical: https://ross.abutalabs.com/products/trafilatura
Homepage: https://trafilatura.readthedocs.io
Language: Python
License: Apache-2.0
License Family: permissive
Topics: web-scraping, text-extraction, nlp, html2text, text-mining, crawler, text-cleaning, text-preprocessing, article-extractor, readability, scraping, news-crawler, tei, html-to-markdown, corpus-builder, corpus-tools, rss-feed, news-aggregator, rag, llm
Last push: 2026-08-21T16:08:46+00:00

## Health v2 (maintenance only)
Score: 93/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 98, release rhythm 83, longevity 100
- inputs: {"age_days": 2704, "days_push": 12, "days_rel": 33, "gap_med": 84, "n_releases_24m": 4}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 6709, forks 421 (observed 2026-08-28T04:09:47.944470+00:00)

## What it is
Trafilatura is a Python package and command-line tool for crawling the web and extracting main text, metadata, and comments from raw HTML while filtering out boilerplate noise. It outputs structured results in formats such as CSV, JSON, HTML, Markdown, TXT, and XML.

## Use cases
- extract article text from web pages
- scrape news articles and metadata
- build text corpora for NLP or LLM training
- convert HTML to markdown or plain text
- crawl sitemaps and RSS feeds for content discovery
- prepare clean web text for RAG pipelines

## When to choose
- you need high-precision main-content extraction from HTML without boilerplate
- you want an all-in-one crawl-download-extract pipeline in Python
- you need multiple output formats including JSON, Markdown, and XML
- you are building datasets for LLM or NLP research

## When to avoid
- you need a full browser-based scraper for JavaScript-heavy sites
- you require structured extraction of arbitrary page elements rather than main content
- you need a distributed large-scale crawling framework

## Facets
- artifact type: library
- maturity: active
- function: web-scraping, parser, nlp, cli, http-client, rag
- domain: web-development, crawlers, large-language-models, data-science
- platform: python, cli, cross-platform
- tags: html-to-markdown, text-extraction, readability, article-extraction, corpus-building, metadata-extraction, news-crawler, natural-language-processing

## Member repositories
- adbar/trafilatura (main) score 93

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:09:47.944470+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T17:42:56.825441+00:00, confidence not recorded.
  - readme: https://github.com/adbar/trafilatura (fetched 2026-08-28T04:09:47.944470+00:00, sha 4a0b6de17b52)
  - registry_pypi: https://pypi.org/pypi/trafilatura/json (fetched 2026-08-29T08:39:30.147870+00:00, sha 0d7d14be21ca)
- Data as of 2026-08-30T08:39:29.467469+00:00.