{"name":"requests-html: Pythonic HTML Parsing with JavaScript Support","description":"requests-html is a Python library designed to simplify HTML parsing and web scraping. It extends the familiar Requests experience with powerful parsing capabilities, including full JavaScript support via Chromium, CSS selectors, and XPath. This makes it an ideal tool for developers needing to interact with dynamic web content.","github":"https://github.com/psf/requests-html","url":"https://osrepos.com/repo/psf-requests-html","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/psf-requests-html","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/psf-requests-html.md","json":"https://osrepos.com/repo/psf-requests-html.json","topics":["Python","Web Scraping","HTML Parsing","Requests","JavaScript Rendering","CSS Selectors","XPath","Asynchronous"],"keywords":["Python","Web Scraping","HTML Parsing","Requests","JavaScript Rendering","CSS Selectors","XPath","Asynchronous"],"stars":null,"summary":"requests-html is a Python library designed to simplify HTML parsing and web scraping. It extends the familiar Requests experience with powerful parsing capabilities, including full JavaScript support via Chromium, CSS selectors, and XPath. This makes it an ideal tool for developers needing to interact with dynamic web content.","content":"## Introduction\n\n`requests-html` is a modern, Pythonic library for parsing HTML and scraping websites, created by Kenneth Reitz, the author of the popular `Requests` library. It aims to make web scraping as intuitive as possible by integrating robust features directly into the `Requests` workflow. Key capabilities include full JavaScript support, CSS and XPath selectors, and asynchronous operations.\n\n## Installation\n\nInstallation is straightforward. You can install `requests-html` using `pip`:\n\nshell\n$ pip install requests-html\n\n\nIf you prefer `pipenv`, you can use:\n\nshell\n$ pipenv install requests-html\n\n\nNote that `requests-html` requires Python 3.6 and above.\n\n## Examples\n\n### Basic GET Request and Link Extraction\n\npython\nfrom requests_html import HTMLSession\n\nsession = HTMLSession()\nr = session.get('https://python.org/')\n\nprint(\"All links:\", r.html.links)\nprint(\"Absolute links:\", r.html.absolute_links)\n\n\n### Using CSS Selectors\n\npython\nfrom requests_html import HTMLSession\n\nsession = HTMLSession()\nr = session.get('https://python.org/')\n\nabout_element = r.html.find('#about', first=True)\nif about_element:\n    print(\"Element text:\", about_element.text)\n    print(\"Element HTML:\", about_element.html)\n    print(\"Element attributes:\", about_element.attrs)\n\n\n### Handling JavaScript Rendered Content\n\n`requests-html` can render JavaScript content using Chromium. The first time `render()` is called, it will download Chromium.\n\npython\nfrom requests_html import HTMLSession\n\nsession = HTMLSession()\nr = session.get('https://pythonclock.org')\n\n# Content before rendering JavaScript\nprint(\"Before render:\", r.html.search('Python 2.7 will retire in...{}Enable Guido Mode')[0])\n\n# Render JavaScript\nr.html.render()\n\n# Content after rendering JavaScript\nprint(\"After render:\", r.html.search('Python 2.7 will retire in...{}Enable Guido Mode')[0])\n\n# Extract specific data after rendering\nperiods = [element.text for element in r.html.find('.countdown-period')]\namounts = [element.text for element in r.html.find('.countdown-amount')]\ncountdown_data = dict(zip(periods, amounts))\nprint(\"Countdown data:\", countdown_data)\n\n\n### Asynchronous Requests\n\nFor concurrent scraping, `requests-html` supports asynchronous operations.\n\npython\nfrom requests_html import AsyncHTMLSession\n\nasession = AsyncHTMLSession()\n\nasync def get_pythonorg():\n    r = await asession.get('https://python.org/')\n    return r\n\nasync def get_reddit():\n    r = await asession.get('https://reddit.com/')\n    return r\n\nresults = asession.run(get_pythonorg, get_reddit)\nfor result in results:\n    print(result.html.url)\n\n\n## Why Use It\n\n`requests-html` stands out by offering a comprehensive solution for web scraping. Its seamless integration with the `Requests` library makes it immediately familiar to Python developers. The built-in JavaScript rendering capability, powered by Chromium, is a significant advantage for modern, dynamic websites. Furthermore, the support for both CSS selectors (jQuery-style) and XPath provides flexibility in element selection, making it a powerful and versatile tool for extracting data from the web.\n\n## Links\n\n*   GitHub Repository: [https://github.com/psf/requests-html](https://github.com/psf/requests-html)","metrics":{"detailViews":1,"githubClicks":1},"dates":{"published":null,"modified":"2026-07-25T08:42:51.000Z"}}