sitefetch: Turn Website Content Into a Text File

Overview

sitefetch is a command-line tool and TypeScript library for collecting readable content from across a website and saving it as a text file. It addresses the tedious work of gathering pages individually when preparing site content for AI models or other text-based workflows.

It uses Mozilla Readability to extract page content, supports limiting crawls to matching URL paths, and lets you specify a CSS selector when the default extraction is not suitable. Use it when you want a local text output from a site rather than a one-page summary.

Key Features

• Fetches pages across a website and writes the extracted content to a text file.
• Accepts URL path patterns to include selected sections, such as blog posts or guides.
• Supports configurable concurrency for fetching pages.
• Extracts readable content with Mozilla Readability.
• Allows a custom CSS content selector when automatic extraction is unsuitable.
• Offers both a command-line interface and a fetchSite TypeScript API.

Use Cases

• Developers can collect documentation from a site to provide as context to an AI model.
• Researchers can gather readable text from selected website sections for later analysis.
• Technical writers can create a text-based working copy of a site's guides or blog posts.
• Teams can automate site-content collection in scripts using the TypeScript API.

Getting Started

Run it once with npm:

npx sitefetch https://example.com -o site.txt

See the README for matching patterns, content selectors, installation options, and API details.

Alternatives

• trafilatura: Trafilatura focuses on extracting text and metadata from HTML, while sitefetch crawls a site and saves readable page content to a text file.
• AnyCrawl: AnyCrawl is a broader toolkit for batch scraping and LLM-based structured extraction, while sitefetch targets readable site content in a text file.
• firecrawl: Firecrawl offers web crawling and extraction through an API or self-hosted deployment, while sitefetch is a CLI for saving readable pages to a text file.
• deepscrape: DeepScrape is a self-hosted service with APIs, browser rendering, and optional LLM extraction, while sitefetch is a CLI for text-file output.

Considerations

• The README does not describe authentication or handling of sites that require login, so do not assume it can crawl private content.
• Readability may extract irrelevant content on some pages. A CSS content selector can help, but may require site-specific tuning.
• Crawling an entire site can fetch many pages. Use path matching to narrow the scope and adjust concurrency as appropriate.