Repository History
6 repositories tagged with web scraping
Lassie: Web Content Retrieval for Humans
Lassie is a powerful Python library designed for efficient web content retrieval. It simplifies the process of extracting essential information like titles, descriptions, images, and videos from various web pages. This tool is ideal for developers needing to programmatically fetch and parse web content with ease.
toapi: Declaratively Turn Any Website into a JSON API
toapi is a powerful Python library designed to transform any website into a clean JSON API declaratively. It enables users to define desired data fields using CSS selectors, fetching and parsing web pages on demand. With built-in caching and support for dynamic content, toapi simplifies web data extraction without complex crawlers or databases.
MechanicalSoup: A Python Library for Automating Website Interaction
MechanicalSoup is a powerful Python library designed for automating interactions with websites. Built upon Requests and BeautifulSoup, it simplifies tasks like storing cookies, following redirects, and submitting forms. It's an excellent tool for web automation tasks that don't require JavaScript execution.
DeepScrape: Intelligent Web Scraping & LLM-Powered Data Extraction
DeepScrape is an AI-powered web scraping tool designed for intelligent data extraction using LLMs. It leverages Playwright for browser automation and supports both cloud (OpenAI) and local LLMs (Ollama, vLLM) for transforming web content into structured JSON. This versatile tool is ideal for modern web applications, RAG pipelines, and various data workflows, offering privacy-first data processing.
Newspaper3k: Advanced News and Article Extraction in Python
Newspaper3k is a powerful Python 3 library designed for news, full-text, and article metadata extraction. Inspired by the simplicity of 'requests' and the speed of 'lxml', it provides robust tools for scraping and curating articles from various sources. This library is ideal for developers needing to programmatically gather and process news content with advanced NLP capabilities.
Pipet: A Swiss-Army Tool for Web Scraping and Data Extraction
Pipet is a versatile command-line web scraper designed for hackers, enabling efficient data extraction from various online assets. It supports HTML parsing, JSON parsing, and client-side JavaScript evaluation, leveraging existing tools like `curl` and `playwright` for powerful and flexible scraping operations. This tool is ideal for tracking information, monitoring changes, and automating data collection tasks.