newspaper: Extract News Articles and Metadata in Python

Overview

Newspaper3k is a Python library for discovering article URLs on known news sites and extracting structured content from their pages. It addresses the repetitive work of downloading HTML and separating article text and metadata from surrounding site content.

Use it when you need a Python-based building block for news collection or analysis. It can also process HTML that your application has already fetched, and supports multiple languages. For a single site or article format that needs highly customized handling, a purpose-built scraper may be a better fit.

Key Features

• Build a publication crawl and identify article and category URLs.
• Download and parse individual articles from URLs.
• Extract article text, title, authors, publication date, and images.
• Generate keywords and a summary from article text using its NLP interface.
• Extract full text directly from HTML with the fulltext helper.
• Support multiple languages, including English, Chinese, German, and Arabic.
• Download articles from multiple sources using a threaded pool.

Use Cases

• News aggregators can collect article links and content from a set of known publications.
• Researchers and analysts can gather article text and metadata for downstream review or processing.
• Monitoring tools can extract article titles and summaries from selected news sources.
• Python developers can turn fetched HTML into readable text without building an article parser from scratch.

Getting Started

Install the Python 3 package:

pip3 install newspaper3k

The README describes additional system dependencies and NLP corpus setup. See the README and documentation for details.

Alternatives

• Trafilatura: Trafilatura focuses on extracting clean text and metadata from web pages, while newspaper also targets news articles and provides images, keywords, and summaries.
• Lassie: Lassie extracts general page metadata and media such as titles, descriptions, images, and videos, rather than focusing on news article text and NLP features.

Considerations

• The project targets Python 3. The README identifies its Python 2 branch as deprecated and buggy.
• Installation may require system libraries for lxml and image handling. NLP-related corpora may also need to be downloaded separately.
• Extraction quality can depend on the structure of the source page. Validate results against the publications you intend to process.
• Crawling sites at scale can encounter rate limits or blocks. Configure request behavior responsibly and follow each site's access policies.