Awesome-crawler: A Curated List of Web Crawlers and Spiders
This repository profile is provided by osrepos.com, an open source repository discovery platform.
Summary
Awesome-crawler is an extensive GitHub repository that curates a collection of web crawling and scraping tools across various programming languages. This resource is invaluable for developers looking for efficient solutions to extract data from the web. It provides a comprehensive overview of popular frameworks and libraries, making it easier to choose the right tool for any web scraping project.
Repository Information
Topics
Click on any tag to explore related repositories
Use at your own risk
OSRepos shares public repositories for knowledge and discovery only. Any installation, execution, configuration, or use of code from these repositories is the user's own responsibility. Always review the repository, source code, dependencies, licenses, and security implications before running or installing anything. OSRepos is not responsible for issues, damages, or losses resulting from third-party repositories.
Introduction
The awesome-crawler repository, maintained by BruceDone, is a highly starred and forked collection of web crawler and spider projects. It serves as a central hub for discovering tools and frameworks designed for web scraping and data extraction, categorized by programming language. With over 7,000 stars, it's a trusted resource in the web crawling community.
Installation
As awesome-crawler is a curated list, there is no direct 'installation' for the repository itself. To leverage this resource, simply navigate to the GitHub repository and explore the various tools listed. Each entry provides a link to its respective project, where you can find specific installation instructions for that particular crawler or scraper.
Examples
The repository organizes its content by programming language, offering a wide array of examples. For instance, under Python, you'll find popular frameworks like Scrapy and PySpider. The Java section features tools such as Apache Nutch and Crawler4j, while JavaScript includes node-crawler and crawlee. This multi-language approach ensures that developers can find relevant tools regardless of their preferred stack.
Why Use
Using awesome-crawler saves significant time and effort in researching web scraping tools. It provides a comprehensive, categorized, and community-vetted list, ensuring you discover robust and well-maintained projects. Whether you're a beginner looking for a simple scraper or an experienced developer needing a distributed crawling framework, this repository offers a starting point for every need across multiple languages.
Links
- GitHub Repository: https://github.com/BruceDone/awesome-crawler
Related repositories
Similar repositories that may be relevant next.
Awesome AI Coding Sandboxes: A Security-First Guide to Agent Code Execution
September 19, 2026
Awesome AI Coding Sandboxes is a curated list of solutions for securely running code generated by autonomous AI agents. It uniquely ranks these sandboxes based on their security posture, focusing on isolation, egress control, and secrets handling. This resource helps developers choose robust environments to mitigate risks associated with untrusted agent code.
Awesome-Self-Evolving-Agents: A Curated List for AI Agent Research
September 14, 2026
Awesome-Self-Evolving-Agents is a comprehensive GitHub repository offering a curated collection of resources on self-evolving agents. It includes a systematic survey, research papers, benchmarks, and open-source projects, providing valuable insights into this rapidly advancing field of AI. This repository serves as an essential guide for researchers and developers exploring model-centric, environment-centric, and co-evolutionary approaches.
Awesome-Self-Improving-Agents: A Curated List for Agentic AI Self-Improvement
September 14, 2026
Awesome-Self-Improving-Agents is a comprehensive GitHub repository featuring a curated and continuously updated list of resources on self-improvement in foundation model-based agentic systems. It serves as a central hub for researchers and practitioners, offering papers, benchmarks, and various media. This resource is essential for anyone exploring the cutting edge of self-evolving AI agents.
Awesome Automated AI/ML: Your Curated Guide to AI/ML Automation Tools
September 12, 2026
Awesome Automated AI/ML is a comprehensive, curated list featuring over 300 tools for automating various aspects of AI and Machine Learning. It covers everything from hyperparameter optimization to autonomous AI agents, offering a dynamic resource for ML engineers, AI researchers, and product builders.
Source repository
Open the original repository on GitHub.
24 counted GitHub visits