toapi: Turn Web Pages into JSON APIs

Overview

toapi is a Python library for exposing selected data from web pages through JSON APIs. You define source pages, API routes, and fields using CSS selectors, then serve the extracted data through Flask.

It is useful when a small service needs structured data from HTML but a full scraping pipeline would be unnecessary. Pages are fetched and parsed on demand, with caching; a headless browser can be configured for JavaScript-heavy sites.

Key Features

• Define extracted fields with CSS selectors and HTML attributes.
• Map API paths and URL parameters to source pages.
• Combine multiple websites behind one API.
• Transform extracted values with field-specific cleaning methods.
• Cache fetched pages and parsed results automatically.
• Optionally use a headless browser for JavaScript-heavy pages.
• Serve results as JSON using Flask.

Use Cases

• A developer can expose selected news listing fields as JSON for a personal app or integration.
• A small team can create a simple data endpoint from HTML when maintaining a dedicated crawler and database is not justified.
• An application can normalize data from multiple sites behind a single API route structure.
• A developer can prototype HTML data extraction and add cleaning rules before deciding whether a larger scraping system is needed.

Getting Started

Requires Python 3.10 or later. Install with:

pip install toapi

See the README for usage examples and development instructions.

Alternatives

• grab: Grab is a broader Python scraping framework with request handling and concurrency, rather than a lightweight service for defining JSON endpoints with CSS selectors.
• AnyCrawl: AnyCrawl supports site crawling, batches, browser rendering, and LLM-based extraction, while toapi focuses on CSS-selector-driven JSON endpoints.
• deepscrape: DeepScrape is a self-hosted service with browser-based crawling and optional LLM extraction, rather than toapi’s declarative CSS-selector approach.

Considerations

• Extraction depends on page HTML and CSS selectors, so changes to a source site's markup may require updating the declarations.
• The project fetches pages from external websites. Consider each site's access policies and the reliability of its pages before building a dependent service.
• The README describes in-memory caching, which may not suit deployments that need shared or persistent cache storage.
• JavaScript-heavy pages may require configuring a headless browser and its driver.
• Python 3.10 or later is required.