firecrawl: Search, Scrape, and Crawl the Web for AI

Overview

Firecrawl is a web data platform and API for searching, scraping, mapping, and crawling websites. It handles the browser and extraction work needed to turn web content into formats such as Markdown or structured JSON, so developers can feed current web information into AI agents and applications.

It is most useful when collecting web data reliably is more work than the application’s core logic. Firecrawl offers a hosted service and an AGPL-3.0-licensed codebase for teams that want to inspect or self-host the software.

Key Features

• Scrape individual URLs into Markdown, HTML, screenshots, or structured JSON.
• Search the web and retrieve content from results.
• Crawl a site asynchronously and collect content from multiple pages.
• Map a website to discover its URLs.
• Batch scrape multiple URLs.
• Interact with pages using actions such as clicking, scrolling, and typing before extraction.
• Use an agent to gather information from a prompt, optionally with a schema or specified URLs.
• Access the API through SDKs for languages including Python, TypeScript, Go, Java, Rust, and others.

Use Cases

• AI application developers can retrieve page content for retrieval, summarization, or agent workflows when the model needs up-to-date web information.
• Data engineers can crawl or batch-scrape sites to build datasets and recurring content pipelines.
• Researchers and analysts can search for sources and extract comparable information across pages without manually browsing each site.
• Product teams can connect web data gathering to MCP-compatible clients or agent tools when their workflows need browsing and page interaction.

Getting Started

Get an API key from firecrawl.dev, then install the Python SDK with pip install firecrawl-py. The README includes examples for the API, CLI, and other SDKs, as well as links to self-hosting instructions.

Alternatives

• deepscrape: DeepScrape is a self-hosted scraping service built around HTTP fetching, Playwright, and optional LLM extraction, rather than Firecrawl’s broader web data API.
• AnyCrawl: AnyCrawl focuses on page scraping, site crawling, batch URLs, and LLM extraction, while Firecrawl also offers web search and a broader API.
• brightdata-mcp: Bright Data MCP provides web access through Bright Data’s managed infrastructure and token, rather than a self-hosted crawling service.

Considerations

• Hosted API examples require a Firecrawl API key. Self-hosting is available, but requires following the project’s setup guide.
• The main project is licensed under AGPL-3.0. The README notes that SDKs and some UI components use MIT licenses, so check the relevant license files for your intended use.
• Scraping users are responsible for respecting site policies and applicable privacy terms. The project says it respects robots.txt by default.
• Firecrawl supports many workflows and SDKs, so teams should confirm the specific endpoint and client behavior they need in the documentation.

For full setup and API examples, see the README.