deepscrape: Scrape Websites and Extract Structured Data

Overview

DeepScrape turns web pages and documents into formats suitable for applications, data pipelines, and retrieval-augmented generation (RAG). It offers a fast HTTP path for server-rendered pages and Playwright for JavaScript-heavy sites, with deterministic CSS extraction as well as optional schema-based LLM extraction.

It is aimed at developers who need more than a one-off scraper: the API also supports multi-page crawling, asynchronous jobs, interactive browser sessions, and an MCP server for AI tools. Its breadth makes it useful as a self-hosted scraping service, though it requires operational setup and is not a way to bypass CAPTCHAs or every site's access controls.

Key Features

• Scrape pages into Markdown, HTML, text, links, screenshots, tables, and other formats.
• Extract structured records with CSS selectors, or use an LLM with a supplied schema.
• Crawl multiple pages with URL discovery, filtering, progress reporting, and downloadable results.
• Use Playwright for interactive or JavaScript-rendered pages, or prefer HTTP for suitable static pages.
• Run background jobs with BullMQ and Redis, and scale web and worker processes separately.
• Connect agent workflows through the included MCP server and typed Node.js SDK.
• Apply SSRF protections, API-key authentication, rate limits, and per-key quotas.

Use Cases

• Build a documentation or knowledge-base corpus for a RAG pipeline, using crawling and Markdown output.
• Collect consistent product, article, or listing data with CSS selectors when predictable output matters more than LLM flexibility.
• Extract fields from pages whose layouts vary, using schema-based LLM extraction and its confidence signals.
• Give an AI agent access to web scraping, search, crawling, or browser sessions through MCP.
• Monitor pages for content changes or turn recurring extractions into saved, self-healing site specifications.

Getting Started

git clone https://github.com/stretchcloud/deepscrape.git
cd deepscrape
npm install
cp .env.example .env
npm run dev

Configure the API key and any required services in .env. Redis is used for queued jobs, and LLM extraction requires an OpenAI-compatible endpoint. See the README for configuration and API details.

Alternatives

• Firecrawl: Firecrawl offers a hosted-style API for search, scraping, and crawling, while DeepScrape is a self-hosted TypeScript service with optional LLM extraction.
• Scraperr: Scraperr focuses on no-code XPath extraction, queues, and exports, rather than DeepScrape’s API-oriented TypeScript workflows and optional LLM extraction.
• toapi: toapi declaratively turns pages into JSON APIs using CSS selectors, while DeepScrape provides broader crawling and scraping through a self-hosted service.
• AnyCrawl: AnyCrawl emphasizes high-throughput Node.js crawling and SERP extraction, while DeepScrape combines HTTP and Playwright fetching with optional LLM extraction.

Considerations

• This is a self-hosted service, not a hosted scraping API. Deployment may involve Redis, Playwright browser dependencies, and environment configuration.
• LLM-based extraction and agent tasks need an OpenAI-compatible model endpoint; deterministic CSS extraction does not require an LLM.
• The README states that fingerprint hygiene does not solve CAPTCHAs or defeat services such as Cloudflare Turnstile and DataDome. Crawling honors robots.txt by default.
• Interactive browser sessions are in memory and pinned to one process, so multi-replica deployments may need sticky routing.
• The supplied repository metadata reports zero open issues, but that alone does not establish maintenance quality or production readiness.