Introduction

Trafilatura is a cutting-edge Python package and command-line tool designed to gather text and metadata from the Web. It simplifies the process of turning raw HTML into structured, meaningful data, offering essential components for web crawling, downloads, scraping, and extraction of main texts, metadata, and comments. Recognized for its robustness and speed, Trafilatura is widely used by companies like HuggingFace, IBM, and Microsoft Research, as well as by renowned academic institutions.

Installation

Installing Trafilatura is straightforward and can be done using pip:

pip install trafilatura

Examples

Here's how you can use Trafilatura to extract text from a URL or an HTML string:

import trafilatura
from trafilatura.downloads import fetch_url

# Example 1: Extract from a URL
url = "https://adrien.barbaresi.eu/blog/trafilatura-web-scraping.html"
print(f"Extracting from: {url}")
downloaded = fetch_url(url)
if downloaded:
    text = trafilatura.extract(downloaded)
    print("--- Extracted Content (first 500 characters) ---")
    print(text[:500])
    print("----------------------------------------------------")

# Example 2: Extract from an HTML string
html_content = """

    
    
        
Main Title
        
This is an example paragraph with <b>bold text</b>.
        

            
Item 1
            
Item 2
        
    

"""
print("\nExtracting from an HTML string:")
text_from_html = trafilatura.extract(html_content)
print("--- Extracted Content ---")
print(text_from_html)
print("-------------------------")

Why Use Trafilatura?

Trafilatura stands out for several reasons, making it an excellent choice for your web content extraction needs:

• Efficiency and Accuracy: It consistently outperforms other open-source libraries in text extraction benchmarks, balancing noise limitation with the inclusion of all valid parts.
• Comprehensive Features: It offers advanced web crawling (sitemaps and feeds support), parallel processing, and robust extraction of main text, metadata (title, author, date), and formatting (paragraphs, titles, lists).
• Multiple Output Formats: It supports TXT, Markdown, CSV, JSON, HTML, XML, and TEI, providing flexibility for various applications.
• Modularity and Ease of Use: No database is required, and it's designed to be handy and modular, facilitating integration into your projects.
• Active Maintenance and Community Support: It receives regular updates, feature additions, and optimizations, backed by comprehensive documentation and an active community.
• Focus on Content Quality: It helps focus on the actual content, avoiding noise from recurring elements like headers and footers, and making sense of data and metadata.

Links

• GitHub Repository: https://github.com/adbar/trafilatura
• Official Documentation: https://trafilatura.readthedocs.io/
• PyPI Package: https://pypi.org/project/trafilatura/