pipet: Scrape and Extract Data from the Web

Overview

Pipet turns web data extraction into a set of reusable text-file queries. It can fetch resources with curl, parse HTML using CSS selectors, query JSON with GJSON syntax, or evaluate JavaScript in a Playwright browser. Results can be structured as text or JSON and passed through Unix tools.

It is most useful when you want a small, scriptable way to collect information without building a dedicated scraper application. Its command-line workflow favors users comfortable with selectors, shell commands, and editing configuration files.

Key Features

• Extract HTML content with nested CSS selector queries.
• Select values from JSON responses using GJSON syntax.
• Run JavaScript against pages loaded in a headless Playwright browser.
• Process query results with Unix pipes and external tools.
• Output results as JSON, customize text separators, or render results with a template.
• Follow next-page links for HTML scraping, with a configurable maximum page count.
• Rerun a query on an interval and execute a command when results change.
• Use copied cURL requests, including headers and cookies, for access to data behind login pages or bot checks.

Use Cases

• A developer can collect headlines, links, or other structured page content for a personal script.
• A user can watch a page for changes, such as a ticket listing becoming available, and trigger a notification command.
• Someone working with an API can select fields from JSON and pass them through jq or other shell tools.
• A researcher or hobbyist can periodically gather public information such as weather, transit departures, or shipment status.

Getting Started

Install with Go and run a query file:

go install github.com/bjesus/pipet/cmd/pipet@latest
pipet hackernews.pipet

See the README for query syntax, installation options, and examples.

Alternatives

• Scrapling: Scrapling is a Python scraping library with adaptive selectors and stealth features, while pipet is a Go command-line tool for reusable extraction queries.
• sitefetch: sitefetch saves matched website pages as plain text, while pipet extracts selected data from HTML, JSON endpoints, and JavaScript-rendered sites.

Considerations

Pipet is a command-line tool rather than a managed scraping service. It relies on external command-line tools such as curl and supports Unix pipes; JavaScript-rendered pages require a Playwright browser, which Pipet may download if one is not installed. Web page structure and access policies can change, so selectors and requests may need maintenance.