requests-html: Fetch and Parse HTML in Python

Overview

requests-html is a Python library for fetching web pages and extracting information from their HTML. It brings HTTP requests and common parsing tasks together, so Python developers can select elements, inspect attributes, find links, and search page text without assembling those steps from separate APIs.

It also offers optional browser-based rendering for pages whose content is created by JavaScript. The project is most relevant for small scraping tasks and scripts that benefit from a compact Requests-like interface; rendering is not necessary for static HTML.

Key Features

• Makes HTTP requests through a Requests-style session.
• Selects elements using CSS selectors or XPath.
• Extracts page links, including absolute links, and reads element text, attributes, and HTML.
• Searches page text using pattern-style queries.
• Provides asynchronous request support for fetching multiple pages.
• Can render JavaScript-generated content with Chromium through pyppeteer.
• Can parse an HTML string directly without making a request.

Use Cases

• A Python developer can extract links or selected fields from static pages for a lightweight scraping script.
• A researcher can collect structured text from several pages, using asynchronous requests when fetching them concurrently is useful.
• A scraper author can inspect content that appears only after JavaScript runs, using browser rendering when ordinary HTML responses are insufficient.
• A developer can parse an HTML fragment already held in memory without setting up an HTTP request.

Getting Started

Install with pipenv:

pipenv install requests-html

See the README for usage examples and details. Project homepage: html.python-requests.org.

Alternatives

• grab: Grab pairs HTTP requests and HTML extraction with crawler workflows, task queues, and proxy support; requests-html also offers optional Chromium rendering.
• Scrapling: Scrapling provides fetching, parsing, and crawling with adaptive selectors, while requests-html centers on Requests-style access and optional Chromium rendering.

Considerations

The README states that Python 3.6 and above is supported. JavaScript rendering relies on Chromium via pyppeteer, and the first render downloads Chromium into the user's home directory. Rendering adds a browser-runtime dependency and is only needed for pages with dynamic content. The repository is not marked archived in the supplied project data; the README and metadata do not establish broader compatibility or maintenance guarantees.