requests-html: Pythonic HTML Parsing with JavaScript Support

Introduction

requests-html is a modern, Pythonic library for parsing HTML and scraping websites, created by Kenneth Reitz, the author of the popular Requests library. It aims to make web scraping as intuitive as possible by integrating robust features directly into the Requests workflow. Key capabilities include full JavaScript support, CSS and XPath selectors, and asynchronous operations.

Installation

Installation is straightforward. You can install requests-html using pip:

$ pip install requests-html

If you prefer pipenv, you can use:

$ pipenv install requests-html

Note that requests-html requires Python 3.6 and above.

Examples

Basic GET Request and Link Extraction

from requests_html import HTMLSession

session = HTMLSession()
r = session.get('https://python.org/')

print("All links:", r.html.links)
print("Absolute links:", r.html.absolute_links)

Using CSS Selectors

from requests_html import HTMLSession

session = HTMLSession()
r = session.get('https://python.org/')

about_element = r.html.find('#about', first=True)
if about_element:
    print("Element text:", about_element.text)
    print("Element HTML:", about_element.html)
    print("Element attributes:", about_element.attrs)

Handling JavaScript Rendered Content

requests-html can render JavaScript content using Chromium. The first time render() is called, it will download Chromium.

from requests_html import HTMLSession

session = HTMLSession()
r = session.get('https://pythonclock.org')

# Content before rendering JavaScript
print("Before render:", r.html.search('Python 2.7 will retire in...{}Enable Guido Mode')[0])

# Render JavaScript
r.html.render()

# Content after rendering JavaScript
print("After render:", r.html.search('Python 2.7 will retire in...{}Enable Guido Mode')[0])

# Extract specific data after rendering
periods = [element.text for element in r.html.find('.countdown-period')]
amounts = [element.text for element in r.html.find('.countdown-amount')]
countdown_data = dict(zip(periods, amounts))
print("Countdown data:", countdown_data)

Asynchronous Requests

For concurrent scraping, requests-html supports asynchronous operations.

from requests_html import AsyncHTMLSession

asession = AsyncHTMLSession()

async def get_pythonorg():
    r = await asession.get('https://python.org/')
    return r

async def get_reddit():
    r = await asession.get('https://reddit.com/')
    return r

results = asession.run(get_pythonorg, get_reddit)
for result in results:
    print(result.html.url)

Why Use It

requests-html stands out by offering a comprehensive solution for web scraping. Its seamless integration with the Requests library makes it immediately familiar to Python developers. The built-in JavaScript rendering capability, powered by Chromium, is a significant advantage for modern, dynamic websites. Furthermore, the support for both CSS selectors (jQuery-style) and XPath provides flexibility in element selection, making it a powerful and versatile tool for extracting data from the web.

Links

• GitHub Repository: https://github.com/psf/requests-html