fhamborg/news-please
news-please - an integrated web crawler and information extractor for news that just works observed · 2026-08-28
Health v2 · maintenance only
67/100
- Activity 77
- Release rhythm 35
- Longevity 100
Flags: no_releases
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-03. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 3545
- days_rel: n/a
- days_push: 141
- n_releases_24m: 0
Adoption not part of the score
2482 stars · 458 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded
news-please is an open-source Python news crawler and information extractor that pulls structured article data (headline, lead, main text, authors, date, language, images) from almost any news website using RSS feeds, recursive link crawling, or the Common Crawl news archive. It works out of the box via a CLI or as an embeddable library, storing results as JSON or in PostgreSQL, Elasticsearch, or Redis.
Use cases
- scrape and extract full text from news article URLs
- crawl an entire news website from its root URL or RSS feed
- build a labeled news dataset for NLP or media research
- download and filter articles from the Common Crawl news archive
- extract metadata like author names and publication dates from articles
- store crawled news articles in Elasticsearch, PostgreSQL, or JSON files
- track how a news article changes over time via repeated crawl revisions
When to choose
- you need turnkey news crawling and extraction that works with just a root URL
- you want to mine the Common Crawl news archive with publisher/date filters
- you need structured article attributes (headline, text, authors, dates) rather than raw HTML
- you want a Python library you can embed in your own data pipeline
- you need pluggable storage backends like JSON, PostgreSQL, Elasticsearch, or Redis
When to avoid
- you need general-purpose web scraping beyond news articles (use Scrapy directly)
- target sites rely heavily on client-side JavaScript rendering
- you need a real-time, high-throughput streaming crawl system
- you only need RSS feed aggregation without full-text extraction
Facets
library · maturity active
web-scraping parser nlp crawlers python cli news-crawler news-extraction article-extraction commoncrawl rss scrapy information-extraction news-archive media-monitoring elasticsearch natural-language-processing data-engineering
2 sources
- readme: https://github.com/fhamborg/news-please · fetched 2026-08-28 · 462cc1753e66
- registry_pypi: https://pypi.org/pypi/news-please/json · fetched 2026-08-29 · 3e9dc62197f1
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| fhamborg/news-please | main | 67 |
For agents
markdown · JSON · MCP: product_card(name="fhamborg/news-please")
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem