Open Source Web Scraping Tools
Web scraping is the automated collection of information from websites. It can turn pages, tables, and dynamically loaded content into structured data for research, monitoring, analysis, and other applications. Scraping tools handle tasks such as fetching pages, rendering JavaScript, selecting relevant content, and managing data output, reducing the effort of gathering information by hand. Responsible use includes respecting site terms, privacy requirements, and limits on automated access.
Open source options range from lightweight libraries and command-line utilities to browser automation frameworks, full crawlers, and services that prepare web content for AI systems. When choosing a tool, consider its license, maintenance activity, documentation, language and runtime requirements, and ability to handle the sites and data formats you need. Also assess reliability, scalability, and integration with your existing workflows. These tools are useful to developers, researchers, analysts, and organizations that need repeatable ways to collect public web data.
30 repositories · updated October 4, 2026
browser-rs-mcp: Share Chrome with AI Agents
A Rust MCP server that lets multiple AI agents control separate tabs in one shared Chrome profile. It provides browser automation through Chrome DevTools Protocol without requiring a Node.js runtime.
lassie: Retrieve Structured Content from Websites
Lassie is a Python library that fetches basic content and metadata from web pages. It suits applications that need page titles, descriptions, images, video details, or other metadata without building their own extraction flow.
requests-html: Fetch and Parse HTML in Python
requests-html combines familiar Requests-style HTTP access with HTML parsing tools for scraping. It supports CSS and XPath selection, link extraction, asynchronous requests, and optional Chromium rendering for JavaScript-generated content.
toapi: Turn Web Pages into JSON APIs
toapi converts web pages into JSON endpoints using declarative CSS selectors. It suits developers who need lightweight, on-demand data extraction without maintaining a separate crawler or database.
grab: Build Python Web Scrapers and Crawlers
Grab is a Python framework for making HTTP requests, extracting data from HTML, and building concurrent crawlers. It suits developers who need reusable scraping workflows with request handling, task queues, and proxy support.
MechanicalSoup: Automate Website Interaction in Python
MechanicalSoup is a Python library for scripting website interactions through HTTP sessions and HTML parsing. Use it to follow links, manage cookies, and submit forms when a site does not require JavaScript.
Agent-Reach: Give AI Agents Access to Web Sources
Agent-Reach is a Python CLI that helps AI agents read and search across websites and platforms such as YouTube, Reddit, GitHub, and Twitter. It installs and checks integrations, with some sources available immediately and others requiring login or configuration.
CloakBrowser: Automate Chromium with Modified Fingerprints
CloakBrowser wraps a custom Chromium build with source-level fingerprint changes and Playwright- and Puppeteer-compatible APIs. It is aimed at browser automation and web scraping where standard browser automation is frequently flagged by bot detection.
ai-website-cloner-template: Recreate Websites as Next.js Apps
A template that equips AI coding agents to inspect a website and recreate it as a Next.js application. It suits developers rebuilding sites they own, recovering lost source code, or studying real-world interface implementations.
firecrawl: Search, Scrape, and Crawl the Web for AI
Firecrawl turns web pages and sites into content that AI applications can use, including Markdown and structured data. It suits developers building agents or data pipelines that need web search, extraction, and crawling through an API or self-hosted deployment.
cheerio: Parse and Manipulate HTML and XML
Cheerio is a TypeScript library for parsing HTML and XML and working with documents through a jQuery-style API. Use it to select, traverse, edit, and render markup without relying on a browser DOM.
trafilatura: Extract Text and Metadata from Web Pages
Trafilatura is a Python library and CLI for discovering web content and extracting readable text and metadata from HTML. It suits developers and researchers building crawlers, web corpora, or content-processing pipelines that need structured output.