# fhamborg/news-please

news-please - an integrated web crawler and information extractor for news that just works

Repository: https://github.com/fhamborg/news-please
Canonical: https://ross.abutalabs.com/products/news-please
Language: Python
License: Apache-2.0
License Family: permissive
Topics: news-crawler, news-extractor, crawler, extractor, news, news-websites, elasticsearch, json, python, nlp, data-gathering, news-archive, news-articles, commoncrawl, extract-articles, extract-information, news-scraper, ccnews, cc-news, roberta
Last push: 2026-04-14T21:17:42+00:00

## Health v2 (maintenance only)
Score: 67/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 77, release rhythm 35, longevity 100
- inputs: {"age_days": 3545, "days_push": 141, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 2482, forks 458 (observed 2026-08-28T04:06:55.406856+00:00)

## What it is
news-please is an open-source Python news crawler and information extractor that pulls structured article data (headline, lead, main text, authors, date, language, images) from almost any news website using RSS feeds, recursive link crawling, or the Common Crawl news archive. It works out of the box via a CLI or as an embeddable library, storing results as JSON or in PostgreSQL, Elasticsearch, or Redis.

## Use cases
- scrape and extract full text from news article URLs
- crawl an entire news website from its root URL or RSS feed
- build a labeled news dataset for NLP or media research
- download and filter articles from the Common Crawl news archive
- extract metadata like author names and publication dates from articles
- store crawled news articles in Elasticsearch, PostgreSQL, or JSON files
- track how a news article changes over time via repeated crawl revisions

## When to choose
- you need turnkey news crawling and extraction that works with just a root URL
- you want to mine the Common Crawl news archive with publisher/date filters
- you need structured article attributes (headline, text, authors, dates) rather than raw HTML
- you want a Python library you can embed in your own data pipeline
- you need pluggable storage backends like JSON, PostgreSQL, Elasticsearch, or Redis

## When to avoid
- you need general-purpose web scraping beyond news articles (use Scrapy directly)
- target sites rely heavily on client-side JavaScript rendering
- you need a real-time, high-throughput streaming crawl system
- you only need RSS feed aggregation without full-text extraction

## Facets
- artifact type: library
- maturity: active
- function: web-scraping, parser, nlp
- domain: crawlers
- platform: python, cli
- tags: news-crawler, news-extraction, article-extraction, commoncrawl, rss, scrapy, information-extraction, news-archive, media-monitoring, elasticsearch, natural-language-processing, data-engineering

## Member repositories
- fhamborg/news-please (main) score 67

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:06:55.406856+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T02:28:12.849341+00:00, confidence not recorded.
  - readme: https://github.com/fhamborg/news-please (fetched 2026-08-28T04:06:55.406856+00:00, sha 462cc1753e66)
  - registry_pypi: https://pypi.org/pypi/news-please/json (fetched 2026-08-29T10:10:18.120930+00:00, sha 3e9dc62197f1)
- Data as of 2026-08-30T08:39:29.467469+00:00.