# toapi: Declaratively Turn Any Website into a JSON API

This repository profile is provided by osrepos.com, an open source repository discovery platform.

Source: osrepos.com
Repository profile: https://osrepos.com/repo/gaojiuli-toapi
Generated for open source discovery and AI-assisted research.

toapi is a powerful Python library designed to transform any website into a clean JSON API declaratively. It enables users to define desired data fields using CSS selectors, fetching and parsing web pages on demand. With built-in caching and support for dynamic content, toapi simplifies web data extraction without complex crawlers or databases.

GitHub: https://github.com/gaojiuli/toapi
OSRepos URL: https://osrepos.com/repo/gaojiuli-toapi

## Summary

toapi is a powerful Python library designed to transform any website into a clean JSON API declaratively. It enables users to define desired data fields using CSS selectors, fetching and parsing web pages on demand. With built-in caching and support for dynamic content, toapi simplifies web data extraction without complex crawlers or databases.

## Topics

- api
- python
- web scraping
- data extraction
- json
- crawler
- flask
- html

## Repository Information

Last analyzed by OSRepos: Fri Jul 24 2026 20:47:26 GMT+0100 (Western European Summer Time)
Detail views: 3
GitHub clicks: 0

## Safety Notice

OSRepos shares public repositories for knowledge and discovery only. Review source code, dependencies, licenses, and security implications before running or installing anything.

## Content

## Introduction

toapi is a powerful Python library designed to declaratively turn any website into a clean JSON API. It simplifies the process of extracting data from web pages by allowing you to define the fields you want using standard CSS selectors. With toapi, pages are fetched and parsed on demand, complete with built-in caching, eliminating the need for complex web crawlers or database maintenance. The core idea behind toapi is that "Every web site provides APIs," and it helps you unlock them.

## Installation

Getting started with toapi is straightforward. You can install it using pip:

bash
pip install toapi


Please note that toapi requires Python 3.10 or higher.

## Examples

Here's a quick example demonstrating how to use toapi to create an API for Hacker News posts:

python
from htmlparsing import Attr, Text
from toapi import Api, Item

api = Api()


@api.site("https://news.ycombinator.com")
@api.list(".athing")
@api.route("/posts", "/news")
@api.route("/posts?page={page}", "/news?p={page}")
class Post(Item):
    title = Text(".titleline > a")
    url = Attr(".titleline > a", "href")


api.run(host="127.0.0.1", port=5000)


After running this Python script, you can visit `http://127.0.0.1:5000/posts` in your browser to get a JSON output similar to this:


{
  "Post": [
    {"title": "Mathematicians Crack the Cursed Curve", "url": "https://www.quantamagazine.org/..."},
    {"title": "Stuffing a Tesla Drivetrain into a 1981 Honda Accord", "url": "https://jalopnik.com/..."}
  ]
}


This example showcases how easily you can define an API endpoint, specify the target website, and extract specific elements using CSS selectors.

## Why Use toapi?

toapi offers several compelling features that make it an excellent choice for web data extraction:

*   **Declarative**: Describe the data you want, not the scraping logic, leading to cleaner and more maintainable code.
*   **Routes**: Map clean API paths to messy source URLs, supporting `{param}` placeholders for dynamic content.
*   **Multi-site Support**: Merge data from several different websites behind a single, unified API.
*   **Cleaning Hooks**: Define `clean_<field>` methods on your `Item` classes to post-process and transform extracted values before they are returned.
*   **Automatic Caching**: Pages and parsed results are automatically cached, improving performance and reducing requests to the source website.
*   **Headless Browser Support**: Easily integrate a headless browser (e.g., via `Api(browser="/path/to/geckodriver")`) for scraping JavaScript-heavy sites.

## Links

*   [toapi GitHub Repository](https://github.com/elliotgao2/toapi)
*   [toapi on PyPI](https://pypi.org/project/toapi/)