{"name":"pagesift: Scrape Web Pages and Parse Documents in Rust","description":"PageSift is a self-hosted Rust tool for turning URLs and documents into clean Markdown or JSON for AI agents. It combines a CLI, Firecrawl- and Jina-compatible APIs, an MCP server, and SDKs, and runs on Linux, macOS, and Windows.","github":"https://github.com/YOLOW92/pagesift","url":"https://osrepos.com/repo/yolow92-pagesift","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/yolow92-pagesift","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/yolow92-pagesift.md","json":"https://osrepos.com/repo/yolow92-pagesift.json","topics":["rust","self-hosted","mcp-server","web-scraping","data-extraction","document-parsing","pdf-extraction"],"keywords":["rust","self-hosted","mcp-server","web-scraping","data-extraction","document-parsing","pdf-extraction"],"stars":null,"summary":"PageSift is a self-hosted Rust tool for turning URLs and documents into clean Markdown or JSON for AI agents. It combines a CLI, Firecrawl- and Jina-compatible APIs, an MCP server, and SDKs, and runs on Linux, macOS, and Windows.","content":"## Overview\n\nPageSift is a self-hosted web scraper, crawler, and document parser built in Rust. It fetches web pages, rendering JavaScript when needed, or reads local documents, then returns Markdown or structured JSON for AI agents, RAG systems, and other software.\n\nIt packages a CLI, Firecrawl- and Jina-compatible HTTP APIs, an MCP server, and client libraries around one Rust binary. It supports Linux, macOS, and Windows, and handles web content alongside PDFs, Office files, and many other document types.\n\n## Key Features\n\n- Scrape individual URLs or crawl sites, with robots.txt handling, per-host limits, and query-focused crawling.\n- Parse PDFs, including scanned pages with optional OCR, plus Office files, e-books, email, archives, and source code.\n- Return Markdown, HTML, links, metadata, or schema-based JSON.\n- Offer Firecrawl-compatible v1/v2 APIs and Jina Reader-style routes for existing integrations.\n- Expose scraping and crawling tools through an MCP server, with an agent endpoint available when configured with a model.\n- Provide browser interactions, change monitoring, and multi-server gateway and worker roles.\n- Distribute Python, Node.js, Rust, and other language bindings, as well as a Docker image.\n\n## Use Cases\n\n- AI application developers can retrieve web pages and documents as Markdown for model context or RAG indexing.\n- Teams replacing a hosted scraping API can run a Firecrawl- or Jina-compatible service on their own infrastructure.\n- Research and data teams can crawl sites or extract fields into JSON using a schema.\n- Agent developers can connect scraping and crawling tools through MCP, or use browser actions for pages that require interaction.\n- Organizations processing mixed document collections can handle PDFs, scans, Office files, and email through one service.\n\n## Our Take\n\nPageSift is a feature-rich, fast-moving Rust extraction platform, but its short history and highly concentrated development call for a measured production rollout.\n\n**Good fit if:**\n- You need a self-hosted Rust service that combines web scraping with broad document parsing.\n- Firecrawl- or Jina-compatible endpoints, MCP, or client SDKs fit your existing agent stack.\n- You can evaluate a young project whose recent development is active but largely driven by one contributor.\n\n**Look elsewhere if:**\n- You need a managed cloud service rather than something to run yourself.\n- Standalone image OCR is essential, which the project says it does not support.\n- AGPL-3.0 obligations do not fit your deployment and a commercial license is not an option.\n\n## Project Health\n\n| Signal | Value |\n|---|---|\n| Status | **New**: less than six months old and changing fast |\n| Last commit | 2026-11-04 (today) |\n| Commits, last 90 days | 100+ |\n| Releases, last 12 months | 30 (latest v1.1.2, 2026-11-04) |\n| Contributors | 2 (top contributor: 99% of commits) |\n| Issues closed, last 90 days | 2 |\n| Pull requests merged, last 90 days | 1 |\n| Project age | 15 days |\n\nChecked on 2026-11-05 with the GitHub API.\n\n## Project Facts\n\n- Language: Rust\n- License: AGPL-3.0\n- Stars: 227\n- Forks: 30\n- Topics: ai-agents, browser-automation, crawl4ai-alternative, data-extraction, document-parsing, firecrawl, firecrawl-alternative, html-to-markdown\n- Archived: no\n\n## What You Need\n\nDetected in the repository:\n\n- Rust 1.89 or newer (from Cargo.toml)\n- A Dockerfile, so it can run in a container\n- Automated checks on GitHub Actions\n\n## Getting Started\n\nInstall the CLI with Cargo and scrape a page:\n\n```bash\ncargo install pagesift --locked\npagesift scrape https://example.com\n```\n\nSee the [README](https://github.com/YOLOW92/pagesift) for server, SDK, MCP, and document-parsing instructions.\n\n## License in Plain Words\n\n**AGPL-3.0** (network copyleft).\n\n- Commercial use: yes\n- Modify and redistribute: yes\n- You must keep: the license notice\n- Share your changes: yes, the full source, also when people only use it over a network (e.g. as a hosted service)\n- Includes an explicit patent grant from the contributors\n\nA summary, not legal advice: the LICENSE file is what applies.\n\n## Alternatives\n\n- [deepscrape](https://osrepos.com/repo/stretchcloud-deepscrape): DeepScrape focuses on crawling websites and scraping structured data, while PageSift also converts documents and offers compatible APIs and an MCP server.\n- [xberg](https://osrepos.com/repo/goldziher-kreuzberg): Xberg focuses on extracting structured content from documents across many formats, while PageSift also handles URLs and provides web-scraping APIs.\n- [docling](https://osrepos.com/repo/docling-project-docling): Docling focuses on local processing of PDFs and other documents, while PageSift also extracts content from URLs and offers a CLI, APIs, and MCP server.\n\n| Project | Language | License | Stars | Status |\n|---|---|---|---|---|\n| **pagesift** | Rust | AGPL-3.0 | 227 | New |\n| [deepscrape](https://osrepos.com/repo/stretchcloud-deepscrape) | TypeScript | MIT | 320 | Active |\n| [xberg](https://osrepos.com/repo/goldziher-kreuzberg) | Rust | MIT | 9.4k | Active |\n| [docling](https://osrepos.com/repo/docling-project-docling) | Python | MIT | 69.8k | Active |\n\n## Considerations\n\nPageSift is very new and changes quickly. Recent commit and release activity is strong, but the project has only two contributors and almost all contribution activity is attributed to one, making continuity a factor to assess. The measured health data shows little recent issue and pull-request handling history, so it provides limited evidence about longer-term maintenance practices.\n\nThe core tool is a Rust binary, while JavaScript rendering requires an optional browser. OCR models download when needed, and formula OCR can require substantial memory; an NVIDIA GPU is optional. Review the AGPL-3.0 terms or the commercial licensing option before adopting it in proprietary deployments. The README also identifies standalone image OCR and managed hosting as gaps.","metrics":{"detailViews":6,"githubClicks":3},"dates":{"published":null,"modified":"2026-11-05T11:04:29.000Z"}}