MechanicalSoup: A Python Library for Automating Website Interaction

Introduction

MechanicalSoup is a Python library for automating interaction with websites. It provides a convenient way to navigate web pages, fill out forms, and follow links programmatically. Inspired by the Mechanize library, MechanicalSoup is built on top of robust Python giants: Requests for handling HTTP sessions and BeautifulSoup for parsing and navigating HTML documents. It automatically manages cookies, follows redirects, and handles form submissions, making web automation straightforward for non-JavaScript-heavy sites. Maintained by a dedicated team since 2017, it offers a reliable solution for many web scraping and interaction needs.

Installation

To get started with MechanicalSoup, you can easily install it using pip. The library supports Python 3 and PyPy3.

pip install MechanicalSoup

For the latest development version, you can install directly from GitHub:

pip install git+https://github.com/MechanicalSoup/MechanicalSoup

Examples

MechanicalSoup simplifies common web interaction tasks. Here's an example demonstrating how to use it to perform a search on Qwant and extract the results:

import re
import mechanicalsoup
import html
import urllib.parse

# Connect to Qwant
browser = mechanicalsoup.StatefulBrowser(user_agent='MechanicalSoup')
browser.open("https://lite.qwant.com/")

# Fill-in the search form
browser.select_form('#search-form')
browser["q"] = "MechanicalSoup"
browser.submit_selected()

# Display the results
for link in browser.page.select('.result a'):
    # Qwant shows redirection links, not the actual URL, so extract
    # the actual URL from the redirect link:
    href = link.attrs['href']
    m = re.match(r"^/redirect/[^/]*/(.*)$", href)
    if m:
        href = urllib.parse.unquote(m.group(1))
    print(link.text, '->', href)

This example showcases how to open a URL, select a form, fill in input fields, submit the form, and then parse the results using BeautifulSoup's powerful selectors.

Why Use MechanicalSoup?

MechanicalSoup excels in scenarios where you need to automate interactions with websites that don't heavily rely on client-side JavaScript. Its key advantages include:

• Simplicity: Provides a high-level API for common web tasks.
• Robustness: Built on top of Requests for reliable HTTP handling and BeautifulSoup for efficient HTML parsing.
• Stateful Browsing: Automatically handles cookies and follows redirects, mimicking a real browser's behavior without the overhead of a full browser engine.
• Form Submission: Easily select and submit forms, including handling various input types.
• Pythonic: Integrates seamlessly into Python projects, making web automation accessible to Python developers.

Links

For more detailed information, documentation, and to contribute, please refer to the official resources:

• Official Documentation (opens in a new tab)
• GitHub Repository (opens in a new tab)
• PyPI Page (opens in a new tab)