{"name":"toapi: Declaratively Turn Any Website into a JSON API","description":"toapi is a powerful Python library designed to transform any website into a clean JSON API declaratively. It enables users to define desired data fields using CSS selectors, fetching and parsing web pages on demand. With built-in caching and support for dynamic content, toapi simplifies web data extraction without complex crawlers or databases.","github":"https://github.com/gaojiuli/toapi","url":"https://osrepos.com/repo/gaojiuli-toapi","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/gaojiuli-toapi","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/gaojiuli-toapi.md","json":"https://osrepos.com/repo/gaojiuli-toapi.json","topics":["api","python","web scraping","data extraction","json","crawler","flask","html"],"keywords":["api","python","web scraping","data extraction","json","crawler","flask","html"],"stars":null,"summary":"toapi is a powerful Python library designed to transform any website into a clean JSON API declaratively. It enables users to define desired data fields using CSS selectors, fetching and parsing web pages on demand. With built-in caching and support for dynamic content, toapi simplifies web data extraction without complex crawlers or databases.","content":"## Introduction\n\ntoapi is a powerful Python library designed to declaratively turn any website into a clean JSON API. It simplifies the process of extracting data from web pages by allowing you to define the fields you want using standard CSS selectors. With toapi, pages are fetched and parsed on demand, complete with built-in caching, eliminating the need for complex web crawlers or database maintenance. The core idea behind toapi is that \"Every web site provides APIs,\" and it helps you unlock them.\n\n## Installation\n\nGetting started with toapi is straightforward. You can install it using pip:\n\nbash\npip install toapi\n\n\nPlease note that toapi requires Python 3.10 or higher.\n\n## Examples\n\nHere's a quick example demonstrating how to use toapi to create an API for Hacker News posts:\n\npython\nfrom htmlparsing import Attr, Text\nfrom toapi import Api, Item\n\napi = Api()\n\n\n@api.site(\"https://news.ycombinator.com\")\n@api.list(\".athing\")\n@api.route(\"/posts\", \"/news\")\n@api.route(\"/posts?page={page}\", \"/news?p={page}\")\nclass Post(Item):\n    title = Text(\".titleline > a\")\n    url = Attr(\".titleline > a\", \"href\")\n\n\napi.run(host=\"127.0.0.1\", port=5000)\n\n\nAfter running this Python script, you can visit `http://127.0.0.1:5000/posts` in your browser to get a JSON output similar to this:\n\n\n{\n  \"Post\": [\n    {\"title\": \"Mathematicians Crack the Cursed Curve\", \"url\": \"https://www.quantamagazine.org/...\"},\n    {\"title\": \"Stuffing a Tesla Drivetrain into a 1981 Honda Accord\", \"url\": \"https://jalopnik.com/...\"}\n  ]\n}\n\n\nThis example showcases how easily you can define an API endpoint, specify the target website, and extract specific elements using CSS selectors.\n\n## Why Use toapi?\n\ntoapi offers several compelling features that make it an excellent choice for web data extraction:\n\n*   **Declarative**: Describe the data you want, not the scraping logic, leading to cleaner and more maintainable code.\n*   **Routes**: Map clean API paths to messy source URLs, supporting `{param}` placeholders for dynamic content.\n*   **Multi-site Support**: Merge data from several different websites behind a single, unified API.\n*   **Cleaning Hooks**: Define `clean_<field>` methods on your `Item` classes to post-process and transform extracted values before they are returned.\n*   **Automatic Caching**: Pages and parsed results are automatically cached, improving performance and reducing requests to the source website.\n*   **Headless Browser Support**: Easily integrate a headless browser (e.g., via `Api(browser=\"/path/to/geckodriver\")`) for scraping JavaScript-heavy sites.\n\n## Links\n\n*   [toapi GitHub Repository](https://github.com/elliotgao2/toapi)\n*   [toapi on PyPI](https://pypi.org/project/toapi/)","metrics":{"detailViews":3,"githubClicks":0},"dates":{"published":null,"modified":"2026-07-24T19:47:26.000Z"}}