# requests-html: Pythonic HTML Parsing with JavaScript Support

This repository profile is provided by osrepos.com, an open source repository discovery platform.

Source: osrepos.com
Repository profile: https://osrepos.com/repo/psf-requests-html
Generated for open source discovery and AI-assisted research.

requests-html is a Python library designed to simplify HTML parsing and web scraping. It extends the familiar Requests experience with powerful parsing capabilities, including full JavaScript support via Chromium, CSS selectors, and XPath. This makes it an ideal tool for developers needing to interact with dynamic web content.

GitHub: https://github.com/psf/requests-html
OSRepos URL: https://osrepos.com/repo/psf-requests-html

## Summary

requests-html is a Python library designed to simplify HTML parsing and web scraping. It extends the familiar Requests experience with powerful parsing capabilities, including full JavaScript support via Chromium, CSS selectors, and XPath. This makes it an ideal tool for developers needing to interact with dynamic web content.

## Topics

- Python
- Web Scraping
- HTML Parsing
- Requests
- JavaScript Rendering
- CSS Selectors
- XPath
- Asynchronous

## Repository Information

Last analyzed by OSRepos: Sat Jul 25 2026 09:42:51 GMT+0100 (Western European Summer Time)
Detail views: 1
GitHub clicks: 1

## Safety Notice

OSRepos shares public repositories for knowledge and discovery only. Review source code, dependencies, licenses, and security implications before running or installing anything.

## Content

## Introduction

`requests-html` is a modern, Pythonic library for parsing HTML and scraping websites, created by Kenneth Reitz, the author of the popular `Requests` library. It aims to make web scraping as intuitive as possible by integrating robust features directly into the `Requests` workflow. Key capabilities include full JavaScript support, CSS and XPath selectors, and asynchronous operations.

## Installation

Installation is straightforward. You can install `requests-html` using `pip`:

shell
$ pip install requests-html


If you prefer `pipenv`, you can use:

shell
$ pipenv install requests-html


Note that `requests-html` requires Python 3.6 and above.

## Examples

### Basic GET Request and Link Extraction

python
from requests_html import HTMLSession

session = HTMLSession()
r = session.get('https://python.org/')

print("All links:", r.html.links)
print("Absolute links:", r.html.absolute_links)


### Using CSS Selectors

python
from requests_html import HTMLSession

session = HTMLSession()
r = session.get('https://python.org/')

about_element = r.html.find('#about', first=True)
if about_element:
    print("Element text:", about_element.text)
    print("Element HTML:", about_element.html)
    print("Element attributes:", about_element.attrs)


### Handling JavaScript Rendered Content

`requests-html` can render JavaScript content using Chromium. The first time `render()` is called, it will download Chromium.

python
from requests_html import HTMLSession

session = HTMLSession()
r = session.get('https://pythonclock.org')

# Content before rendering JavaScript
print("Before render:", r.html.search('Python 2.7 will retire in...{}Enable Guido Mode')[0])

# Render JavaScript
r.html.render()

# Content after rendering JavaScript
print("After render:", r.html.search('Python 2.7 will retire in...{}Enable Guido Mode')[0])

# Extract specific data after rendering
periods = [element.text for element in r.html.find('.countdown-period')]
amounts = [element.text for element in r.html.find('.countdown-amount')]
countdown_data = dict(zip(periods, amounts))
print("Countdown data:", countdown_data)


### Asynchronous Requests

For concurrent scraping, `requests-html` supports asynchronous operations.

python
from requests_html import AsyncHTMLSession

asession = AsyncHTMLSession()

async def get_pythonorg():
    r = await asession.get('https://python.org/')
    return r

async def get_reddit():
    r = await asession.get('https://reddit.com/')
    return r

results = asession.run(get_pythonorg, get_reddit)
for result in results:
    print(result.html.url)


## Why Use It

`requests-html` stands out by offering a comprehensive solution for web scraping. Its seamless integration with the `Requests` library makes it immediately familiar to Python developers. The built-in JavaScript rendering capability, powered by Chromium, is a significant advantage for modern, dynamic websites. Furthermore, the support for both CSS selectors (jQuery-style) and XPath provides flexibility in element selection, making it a powerful and versatile tool for extracting data from the web.

## Links

*   GitHub Repository: [https://github.com/psf/requests-html](https://github.com/psf/requests-html)