AnyCrawl: Crawl Websites and Extract LLM-Ready Data

Overview

AnyCrawl combines web scraping, site traversal, batch processing, and search-result collection in a Node.js service. It addresses the work of collecting website content and preparing it in formats that are easier to use in RAG systems and other LLM workflows.

Teams can use its hosted API or self-host it, with static parsing or browser-based engines for JavaScript-rendered pages. Structured extraction is available through configurable schemas and an LLM provider.

Key Features

• Scrape individual pages and return content in formats such as markdown.
• Crawl sites with depth, page limits, domain scope, and path filters.
• Submit batches of known URLs as asynchronous jobs and retrieve results later.
• Use Cheerio for static pages or Playwright and Puppeteer for browser-rendered content.
• Extract structured JSON from pages using a supplied schema and LLM configuration.
• Configure request-level or self-hosted proxies and caching.
• Run multiple workers or processes for bulk workloads, according to the project description.

Use Cases

• RAG developers can collect and convert documentation or other web pages into LLM-friendly content.
• Data teams can crawl a site or submit a known URL list for recurring ingestion jobs.
• AI application developers can extract fields such as company details from pages into schema-shaped JSON.
• Search and market research workflows can use the search endpoint to collect SERP results where its supported engines meet their needs.

Getting Started

For the hosted API, send a request to the scrape endpoint with an API key:

curl -X POST https://api.anycrawl.dev/v1/scrape \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_ANYCRAWL_API_KEY' \
  -d '{"url":"https://example.com","engine":"cheerio"}'

For self-hosting, deployment and authentication instructions are in the repository README and documentation.

Alternatives

• firecrawl: Firecrawl focuses on AI-ready web search, crawling, and extraction through an API, while AnyCrawl also targets URL batches and SERP scraping.
• deepscrape: DeepScrape is a self-hosted TypeScript service using HTTP fetching and Playwright, with optional LLM extraction; AnyCrawl also supports SERP scraping.
• Scraperr: Scraperr offers a self-hosted, no-code interface with XPath extraction and queued domain spidering, rather than AnyCrawl’s TypeScript toolkit and API focus.
• sitefetch: sitefetch crawls a site and saves readable content to a text file, rather than providing AnyCrawl’s broader API and structured extraction workflows.

Considerations

The project is active, with regular recent commits and releases, and many pull requests merged recently. However, nearly all measured contributor activity comes from one person, and the recent issue-closure measure is zero, so teams should assess support and continuity expectations rather than infer them from release activity alone. The repository reports CI but no tests in its project requirements data.

Self-hosting requires Node.js 20 or newer; Docker and Docker Compose are supported. Browser engines add browser-runtime configuration, and automatic proxy-based GeoIP resolution may download a database on first use. The README also notes that Google is the only supported search engine in its current SERP endpoint documentation. The MIT license is permissive, but distributions should preserve its copyright and license notice.