python-readability: Extract Main Text from HTML Pages

Overview

python-readability is a Python library that identifies and extracts the main body text and title from an HTML document. It is useful when an application needs article content without the surrounding page clutter, such as navigation and unrelated sections.

The project is suited to developers building content-processing workflows. Its output is cleaned-up HTML, not a guarantee that content is safe to render: untrusted output needs a separate HTML sanitizer.

Key Features

• Extracts a document's title and main body content.
• Returns article content as HTML through Document.summary().
• Supports input as HTML content and provides encoding detection for bytes input.
• Includes a command-line interface for processing a local HTML file or URL.
• Offers options and APIs for article metadata and image handling, including authors and keeping multiple images.
• Includes reproducible page-extraction fixtures and benchmark tooling.

Use Cases

• A developer can extract article text before indexing pages for search.
• A content-processing application can present a less cluttered reading view from saved or fetched HTML.
• A researcher can prepare web articles for downstream text analysis, while retaining HTML structure such as headings and links.
• A developer can compare extraction changes against the project's real-page fixtures when maintaining a scraping or publishing pipeline.

Getting Started

Install the package with pip:

pip install readability-lxml

See the README for usage examples and further installation options.

Alternatives

• trafilatura: Trafilatura extracts readable text and metadata for crawlers and research pipelines, while python-readability focuses on an article’s main text and title.
• newspaper: Newspaper3k targets news sites and also extracts images, keywords, and summaries, while python-readability focuses on article text and title.

Considerations

• Extraction is heuristic: page structure and markup can affect which content is selected.
• Document.summary() is not a general-purpose HTML sanitizer or a security boundary. Sanitize untrusted output with a dedicated allowlist sanitizer and use an appropriate Content Security Policy before rendering it.
• The package is licensed under Apache-2.0.