toapi: Declaratively Turn Any Website into a JSON API

Introduction

toapi is a powerful Python library designed to declaratively turn any website into a clean JSON API. It simplifies the process of extracting data from web pages by allowing you to define the fields you want using standard CSS selectors. With toapi, pages are fetched and parsed on demand, complete with built-in caching, eliminating the need for complex web crawlers or database maintenance. The core idea behind toapi is that "Every web site provides APIs," and it helps you unlock them.

Installation

Getting started with toapi is straightforward. You can install it using pip:

pip install toapi

Please note that toapi requires Python 3.10 or higher.

Examples

Here's a quick example demonstrating how to use toapi to create an API for Hacker News posts:

from htmlparsing import Attr, Text
from toapi import Api, Item

api = Api()


@api.site("https://news.ycombinator.com")
@api.list(".athing")
@api.route("/posts", "/news")
@api.route("/posts?page={page}", "/news?p={page}")
class Post(Item):
    title = Text(".titleline > a")
    url = Attr(".titleline > a", "href")


api.run(host="127.0.0.1", port=5000)

After running this Python script, you can visit http://127.0.0.1:5000/posts in your browser to get a JSON output similar to this:

{
  "Post": [
    {"title": "Mathematicians Crack the Cursed Curve", "url": "https://www.quantamagazine.org/..."},
    {"title": "Stuffing a Tesla Drivetrain into a 1981 Honda Accord", "url": "https://jalopnik.com/..."}
  ]
}

This example showcases how easily you can define an API endpoint, specify the target website, and extract specific elements using CSS selectors.

Why Use toapi?

toapi offers several compelling features that make it an excellent choice for web data extraction:

• Declarative: Describe the data you want, not the scraping logic, leading to cleaner and more maintainable code.
• Routes: Map clean API paths to messy source URLs, supporting {param} placeholders for dynamic content.
• Multi-site Support: Merge data from several different websites behind a single, unified API.
• Cleaning Hooks: Define clean_<field> methods on your Item classes to post-process and transform extracted values before they are returned.
• Automatic Caching: Pages and parsed results are automatically cached, improving performance and reducing requests to the source website.
• Headless Browser Support: Easily integrate a headless browser (e.g., via Api(browser="/path/to/geckodriver")) for scraping JavaScript-heavy sites.

Links

• toapi GitHub Repository
• toapi on PyPI