Open Source Web Scraping Tools
Web scraping is the automated collection of information from websites. It can turn pages, tables, and dynamically loaded content into structured data for research, monitoring, analysis, and other applications. Scraping tools handle tasks such as fetching pages, rendering JavaScript, selecting relevant content, and managing data output, reducing the effort of gathering information by hand. Responsible use includes respecting site terms, privacy requirements, and limits on automated access.
Open source options range from lightweight libraries and command-line utilities to browser automation frameworks, full crawlers, and services that prepare web content for AI systems. When choosing a tool, consider its license, maintenance activity, documentation, language and runtime requirements, and ability to handle the sites and data formats you need. Also assess reliability, scalability, and integration with your existing workflows. These tools are useful to developers, researchers, analysts, and organizations that need repeatable ways to collect public web data.
29 repositories · updated October 14, 2026
browser-rs-mcp: Share Chrome with AI Agents
A Rust MCP server that lets multiple AI agents control separate tabs in one shared Chrome profile. It provides browser automation through Chrome DevTools Protocol without requiring a Node.js runtime.
NewHealth 50/100
lassie: Retrieve Structured Content from Websites
Lassie is a Python library that fetches basic content and metadata from web pages. It suits applications that need page titles, descriptions, images, video details, or other metadata without building their own extraction flow.
InactiveHealth 24/100
requests-html: Fetch and Parse HTML in Python
requests-html combines familiar Requests-style HTTP access with HTML parsing tools for scraping. It supports CSS and XPath selection, link extraction, asynchronous requests, and optional Chromium rendering for JavaScript-generated content.
InactiveHealth 30/100
toapi: Turn Web Pages into JSON APIs
toapi converts web pages into JSON endpoints using declarative CSS selectors. It suits developers who need lightweight, on-demand data extraction without maintaining a separate crawler or database.
QuietHealth 28/100
grab: Build Python Web Scrapers and Crawlers
Grab is a Python framework for making HTTP requests, extracting data from HTML, and building concurrent crawlers. It suits developers who need reusable scraping workflows with request handling, task queues, and proxy support.
InactiveHealth 26/100
MechanicalSoup: Automate Website Interaction with Python
MechanicalSoup lets Python programs interact with websites through HTTP sessions and parsed HTML, including following links and submitting forms. It suits scripted web tasks that do not require JavaScript execution.
ActiveHealth 47/100
Agent-Reach: Give AI Agents Access to Web Sources
Agent-Reach is a Python CLI that helps AI agents read and search across websites and platforms such as YouTube, Reddit, GitHub, and Twitter. It installs and checks integrations, with some sources available immediately and others requiring login or configuration.
ActiveHealth 57/100
CloakBrowser: Automate Chromium with Modified Fingerprints
CloakBrowser wraps a custom Chromium build with source-level fingerprint changes and Playwright- and Puppeteer-compatible APIs. It is aimed at browser automation and web scraping where standard browser automation is frequently flagged by bot detection.
ActiveHealth 55/100
ai-website-cloner-template: Recreate Websites as Next.js Apps
A template that equips AI coding agents to inspect a website and recreate it as a Next.js application. It suits developers rebuilding sites they own, recovering lost source code, or studying real-world interface implementations.
ActiveHealth 57/100
firecrawl: Search, Scrape, and Crawl the Web for AI
Firecrawl turns web pages and sites into content that AI applications can use, including Markdown and structured data. It suits developers building agents or data pipelines that need web search, extraction, and crawling through an API or self-hosted deployment.
ActiveHealth 73/100
cheerio: Parse and Manipulate HTML and XML
Cheerio is a TypeScript library for parsing HTML and XML and working with documents through a jQuery-style API. Use it to select, traverse, edit, and render markup without relying on a browser DOM.
ActiveHealth 67/100
trafilatura: Extract Text and Metadata from Web Pages
Trafilatura is a Python library and CLI for discovering web content and extracting readable text and metadata from HTML. It suits developers and researchers building crawlers, web corpora, or content-processing pipelines that need structured output.
ActiveHealth 53/100