trafilatura: Extract Text and Metadata from Web Pages

Overview

Trafilatura turns web pages and HTML files into structured text and metadata, reducing boilerplate such as headers and footers. It combines content discovery, downloading, and extraction in a Python package and command-line tool, with output suited to downstream analysis and storage.

It is useful when a project needs more than plain text conversion, such as article metadata, document structure, or crawl support. It does not require a database, and its configurable extraction can include optional elements such as comments, links, images, and tables.

Key Features

• Discover URLs from sitemaps and ATOM, JSON, or RSS feeds.
• Download and process live URLs, local HTML files, and parsed HTML trees.
• Extract main text and metadata including title, author, date, site name, categories, and tags.
• Preserve structure such as headings, paragraphs, lists, quotes, and code.
• Use its own rule-based extractor, with jusText and readability-lxml as fallbacks.
• Export to TXT, Markdown, CSV, JSON, HTML, XML, and XML-TEI.
• Process input in parallel and manage download queues with filtering and deduplication.

Use Cases

• Researchers can build text corpora from websites and retain metadata for later analysis.
• Data and NLP teams can prepare web content for text-mining, language-processing, or RAG pipelines.
• Developers can extract article text from URLs or saved HTML as part of a content-ingestion workflow.
• News and content services can discover pages through feeds or sitemaps and produce structured records.

Getting Started

Install the package with pip:

pip install trafilatura

For usage options and installation details, see the README and documentation.

Alternatives

• newspaper: Newspaper3k focuses on news articles and also extracts images, keywords, and summaries, while Trafilatura handles a broader range of web pages and structured output.
• Scrapling: Scrapling focuses on general-purpose scraping with adaptive selectors and anti-bot features, while Trafilatura specializes in extracting readable text and metadata.
• pipet: Pipet uses reusable queries to extract selected data from HTML, JSON, and rendered pages, while Trafilatura focuses on readable web text and metadata.

Considerations

Extraction quality can depend on page structure and configuration. The project balances including valid content with removing boilerplate, and offers fallback extractors rather than guaranteeing identical results across sites. Fetching live pages involves network access; for offline workflows, it can also process previously downloaded HTML. Versions before 1.8.0 used GPLv3+, while the current project license is Apache-2.0.