Our take
Newless than six months old and changing fast- Activity
- 82
- Community
- 11
- Issues
- 53
- Pull requests
- 43
- Releases more often than 80% of the projects we track
- Fewer contributors than 86% of the projects we track
PageSift is a feature-rich, fast-moving Rust extraction platform, but its short history and highly concentrated development call for a measured production rollout.
Good fit if
- You need a self-hosted Rust service that combines web scraping with broad document parsing.
- Firecrawl- or Jina-compatible endpoints, MCP, or client SDKs fit your existing agent stack.
- You can evaluate a young project whose recent development is active but largely driven by one contributor.
Look elsewhere if
- You need a managed cloud service rather than something to run yourself.
- Standalone image OCR is essential, which the project says it does not support.
- AGPL-3.0 obligations do not fit your deployment and a commercial license is not an option.
All health signals
| Last commit | 2026-11-04 (today) |
|---|---|
| Commits, last 90 days | 100+ |
| Releases, last 12 months | 30 (latest v1.1.2, 2026-11-04) |
| Contributors | 2 (top contributor: 99% of commits) |
| Issues closed, last 90 days | 2 |
| Pull requests merged, last 90 days | 1 |
| Project age | 15 days |
Checked on 2026-11-05 with the GitHub API.
Overview
PageSift is a self-hosted web scraper, crawler, and document parser built in Rust. It fetches web pages, rendering JavaScript when needed, or reads local documents, then returns Markdown or structured JSON for AI agents, RAG systems, and other software.
It packages a CLI, Firecrawl- and Jina-compatible HTTP APIs, an MCP server, and client libraries around one Rust binary. It supports Linux, macOS, and Windows, and handles web content alongside PDFs, Office files, and many other document types.
Key Features
- Scrape individual URLs or crawl sites, with robots.txt handling, per-host limits, and query-focused crawling.
- Parse PDFs, including scanned pages with optional OCR, plus Office files, e-books, email, archives, and source code.
- Return Markdown, HTML, links, metadata, or schema-based JSON.
- Offer Firecrawl-compatible v1/v2 APIs and Jina Reader-style routes for existing integrations.
- Expose scraping and crawling tools through an MCP server, with an agent endpoint available when configured with a model.
- Provide browser interactions, change monitoring, and multi-server gateway and worker roles.
- Distribute Python, Node.js, Rust, and other language bindings, as well as a Docker image.
Use Cases
- AI application developers can retrieve web pages and documents as Markdown for model context or RAG indexing.
- Teams replacing a hosted scraping API can run a Firecrawl- or Jina-compatible service on their own infrastructure.
- Research and data teams can crawl sites or extract fields into JSON using a schema.
- Agent developers can connect scraping and crawling tools through MCP, or use browser actions for pages that require interaction.
- Organizations processing mixed document collections can handle PDFs, scans, Office files, and email through one service.
What you need
Detected in the repository
- Rust 1.89 or newer (from Cargo.toml)
- A Dockerfile, so it can run in a container
- Automated checks on GitHub Actions
License in plain words
AGPL-3.0network copyleft
- Commercial use: yes
- Modify and redistribute: yes
- You must keep: the license notice
- Share your changes: yes, the full source, also when people only use it over a network (e.g. as a hosted service)
- Includes an explicit patent grant from the contributors:
A summary, not legal advice: the LICENSE file is what applies.
Getting Started
Install the CLI with Cargo and scrape a page:
cargo install pagesift --locked
pagesift scrape https://example.com
See the README for server, SDK, MCP, and document-parsing instructions.
Alternatives
- deepscrape: DeepScrape focuses on crawling websites and scraping structured data, while PageSift also converts documents and offers compatible APIs and an MCP server.
- xberg: Xberg focuses on extracting structured content from documents across many formats, while PageSift also handles URLs and provides web-scraping APIs.
- docling: Docling focuses on local processing of PDFs and other documents, while PageSift also extracts content from URLs and offers a CLI, APIs, and MCP server.
| Project | Language | License | Stars | Status |
|---|---|---|---|---|
| pagesift | Rust | AGPL-3.0 | 227 | New |
| deepscrape | TypeScript | MIT | 320 | Active |
| xberg | Rust | MIT | 9.4k | Active |
| docling | Python | MIT | 69.8k | Active |
Considerations
PageSift is very new and changes quickly. Recent commit and release activity is strong, but the project has only two contributors and almost all contribution activity is attributed to one, making continuity a factor to assess. The measured health data shows little recent issue and pull-request handling history, so it provides limited evidence about longer-term maintenance practices.
The core tool is a Rust binary, while JavaScript rendering requires an optional browser. OCR models download when needed, and formula OCR can require substantial memory; an NVIDIA GPU is optional. Review the AGPL-3.0 terms or the commercial licensing option before adopting it in proprietary deployments. The README also identifies standalone image OCR and managed hosting as gaps.
Found this useful?
Share it with someone who would like pagesift.