Grab: A Powerful Python Web Scraping Framework

Introduction

Grab is a powerful Python web scraping framework designed to streamline the process of making network requests, scraping websites, and processing the extracted content. It offers a rich set of features including automatic cookie management, support for HTTP and SOCKS proxies, and robust tools for interacting with web forms. For advanced asynchronous scraping, Grab includes its Spider component, enabling efficient parallel processing of thousands of queries.

Recently, the Grab project underwent a significant update to ensure compatibility across a wide range of Python versions, from 2.7 up to 3.13. This update, culminating in version 1.0, focuses on stability and modern Python support, making Grab a reliable choice for current development. Please note a backward-incompatible change: DataNotFound and InvalidResponseError exceptions are now located in grab.errors instead of weblib.error.

Installation

Installing Grab is straightforward. You can get started by running the following command:

pip install -U grab

For more detailed instructions and platform-specific guidance, please refer to the official documentation: https://grab.readthedocs.io/en/stable/usage/installation.html

Examples

Grab Example

Here’s a basic example demonstrating how to use Grab to log into a GitHub account and list repositories:

import logging

from grab import Grab

logging.basicConfig(level=logging.DEBUG)

g = Grab()

g.go('https://github.com/login')
g.doc.set_input('login', '****')
g.doc.set_input('password', '****')
g.doc.submit()

g.doc.save('/tmp/x.html')

g.doc('//ul[@id="user-links"]//button[contains(@class, "signout")]').assert_exists()

home_url = g.doc('//a[contains(@class, "header-nav-link name")]/@href').text()
repo_url = home_url + '?tab=repositories'

g.go(repo_url)

for elem in g.doc.select('//h3[@class="repo-list-name"]/a'):
    print('%s: %s' % (elem.text(),
                      g.make_url_absolute(elem.attr('href'))))

Grab::Spider Example

For asynchronous scraping, Grab's Spider component allows you to define tasks and process them in parallel:

import logging

from grab.spider import Spider, Task

logging.basicConfig(level=logging.DEBUG)


class ExampleSpider(Spider):
    def task_generator(self):
        for lang in 'python', 'ruby', 'perl':
            url = 'https://www.google.com/search?q=%s' % lang
            yield Task('search', url=url, lang=lang)

    def task_search(self, grab, task):
        print('%s: %s' % (task.lang,
                          grab.doc('//div[@class="s"]//cite').text()))


bot = ExampleSpider(thread_number=2)
bot.run()

Why Use Grab?

Grab stands out as a versatile web scraping framework due to its comprehensive feature set and robust architecture. It provides:

• Full-featured HTTP Client: Handles cookies, proxies (HTTP/SOCKS), Keep-Alive, and IDN support automatically.
• Powerful Data Extraction: Utilizes XPath queries for precise and efficient data extraction from HTML documents.
• Asynchronous Capabilities: The integrated Spider framework allows for high-performance, parallel scraping, automatically managing network errors and task queues.
• Web Form Interaction: Simplifies working with web forms, including easy multipart file uploads.
• Active Maintenance: Recently updated to support modern Python versions (2.7 to 3.13), ensuring ongoing compatibility and stability.
• Community Support: Access to Telegram chat groups for discussions and assistance.

Links

• GitHub Repository: https://github.com/lorien/grab
• Official Documentation: https://grab.readthedocs.io/en/stable/
• Telegram Chat (English): https://t.me/grablab
• Telegram Chat (Russian): https://t.me/grablab_ru