adbar/trafilatura

Health v2 · maintenance only

Adoption not part of the score

6709 stars · 421 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Trafilatura is a Python package and command-line tool for crawling the web and extracting main text, metadata, and comments from raw HTML while filtering out boilerplate noise. It outputs structured results in formats such as CSV, JSON, HTML, Markdown, TXT, and XML.

Use cases

• extract article text from web pages
• scrape news articles and metadata
• build text corpora for NLP or LLM training
• convert HTML to markdown or plain text
• crawl sitemaps and RSS feeds for content discovery
• prepare clean web text for RAG pipelines

When to choose

• you need high-precision main-content extraction from HTML without boilerplate
• you want an all-in-one crawl-download-extract pipeline in Python
• you need multiple output formats including JSON, Markdown, and XML
• you are building datasets for LLM or NLP research

When to avoid

• you need a full browser-based scraper for JavaScript-heavy sites
• you require structured extraction of arbitrary page elements rather than main content
• you need a distributed large-scale crawling framework

Facets

library · maturity active

web-scraping parser nlp cli http-client rag web-development crawlers large-language-models data-science python cli cross-platform html-to-markdown text-extraction readability article-extraction corpus-building metadata-extraction news-crawler natural-language-processing

Member repositories

For agents

markdown · JSON · MCP: product_card(name="adbar/trafilatura")