Our take
Inactiveno commits in over a year- Activity
- 19
- Community
- 51
- Issues
- 21
- Pull requests
- 20
+38 stars in the last 30 days
This is a broad discovery index, but its lack of recent maintenance means entries should be checked before adoption.
Good fit if
- You want a cross-language starting point for finding crawler and scraper projects.
- You are researching options and will verify each candidate's current status, documentation, and compatibility.
- You value a simple MIT-licensed list rather than an integrated crawling product.
Look elsewhere if
- You need a maintained, tested recommendation set: the repository is marked inactive, with no commits or releases in over a year.
- You need the list itself to provide installation, runtime guidance, or project comparisons, which the README does not supply.
- You need evidence of recent community support: there has been no recent issue closure or pull request merge activity in the measured period.
All health signals
| Last commit | 2024-06-16 (2.4 years ago) |
|---|---|
| Commits, last 90 days | 0 |
| Releases, last 12 months | 0 |
| Contributors | 26 (top contributor: 70% of commits) |
| Issues closed, last 90 days | 0 |
| Pull requests merged, last 90 days | 0 |
| Project age | 10 years |
Checked on 2026-10-31 with the GitHub API.
Project history
Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.
Stars
7,370Forks
751Change since Nov 13, 2026: +3 stars·0 forks
Overview
awesome-crawler is a curated directory of web crawlers, spiders, scraping libraries, and related resources, organized by programming language. It addresses the problem of finding suitable crawling tools across a fragmented ecosystem, rather than providing a crawler of its own.
The list covers Python, Java, C#, JavaScript, PHP, C++, C, Ruby, Rust, R, Erlang, Perl, Go, and Scala. Entries include frameworks, parsers, browser-based tools, distributed systems, and individual utilities, with short descriptions and links to their projects.
Key Features
- Organizes crawler and scraping resources by programming language.
- Includes frameworks for crawling, scraping, and browser interaction.
- Covers distributed and concurrent crawler options, including examples in Python, Java, PHP, and Go.
- Lists HTML and XML parsing and extraction tools alongside full crawlers.
- Includes JavaScript-rendering and headless-browser options.
- Provides links and brief descriptions to help readers investigate individual projects.
Use Cases
- A developer starting a scraper can scan the section for their language and shortlist relevant libraries.
- A team evaluating a distributed crawler can discover options such as Scrapy-Redis, Apache Nutch, or Go-based frameworks.
- A researcher comparing tools across languages can use the language sections as a starting map of the ecosystem.
- Someone who needs browser automation or JavaScript-rendered page support can locate projects explicitly described as supporting those capabilities.
License in plain words
MITpermissive
- Commercial use: yes
- Modify and redistribute: yes
- You must keep: the copyright and license notice
- Share your changes: no
A summary, not legal advice: the LICENSE file is what applies.
Getting Started
Browse the README's language sections, then follow the link for a candidate project. Check that project's own documentation for installation and usage instructions.
Considerations
The repository is inactive, with no commits in over a year and no releases in the past year. Although it has multiple contributors, the measured period shows no issue closures or merged pull requests, so listed projects and links may be stale. The README does not specify runtime requirements, tests, or CI for this collection. The repository uses the MIT license; review the license of each linked project separately before adopting it.
Found this useful?
Share it with someone who would like awesome-crawler.