grab: Build Python Web Scrapers and Crawlers

Overview

Grab combines an HTTP client and HTML extraction API with Spider, a framework for organizing and running scraping tasks. It is aimed at Python developers building site-specific scrapers who want request handling and parsing in one library, with an option to scale work across multiple simultaneous requests.

The repository was reset to the code from the 0.6.41 release and updated for Python 2.7 through Python 3.13. The maintainer describes this as a compatibility and maintenance update, not a source of new features. See the project README and documentation for implementation details.

Key Features

• HTTP requests with session cookies, Keep-Alive, and HTTP or SOCKS proxy support.
• HTML DOM queries using XPath, with helpers for forms and multipart file uploads.
• Automatic charset detection and configurable HTTP requests.
• Spider task generation and parsing organized into separate methods.
• Multiple parallel requests and automatic re-queuing of failed tasks.
• Task queue backends including in-memory storage, Redis, and MongoDB.
• Optional caching of network results and tools for debugging and statistics.

Use Cases

• A Python developer needs to collect structured data from pages and prefers XPath-based extraction over building request and parsing utilities from scratch.
• A scraping team has many URLs to process and wants a task-based crawler with parallel requests and retry handling.
• A workflow needs authenticated sessions, form submission, or proxy configuration while interacting with a site.
• A developer wants to retain crawler tasks or cache responses using a supported Redis or MongoDB backend.

Getting Started

Install with:

pip install -U grab

See the README and installation documentation for details.

Alternatives

• Scrapling: Scrapling focuses on adaptive extraction and anti-bot scraping, while Grab provides reusable request handling, task queues, and proxy support.

Considerations

• The 2025 update restores and modernizes the codebase from the 2018 release; the maintainer explicitly says it adds no new features.
• The update introduces an incompatible exception change: DataNotFound and InvalidResponseError are now in grab.errors rather than the former weblib.error exceptions. Existing imports and exception handling may need changes.
• Users who need to stay on the older release can specify grab<1.0, according to the README.
• The project targets a wide Python range, from 2.7 through 3.13. Check the documentation and your dependency stack for compatibility with your environment.