StormCrawler
| Company type | Open-source software project |
|---|---|
| Industry | Software, Web crawling |
Area served | Worldwide |
| Products | Framework for scalable, low-latency web crawlers |
| Owner | Apache Software Foundation |
| Website | stormcrawler |
Apache StormCrawler is a free and open-source framework designed for constructing scalable, low-latency web crawlers utilizing Apache Storm. Primarily implemented in Java (programming language) and distributed under the Apache License, it features modular elements including a core package that delivers fundamental crawling capabilities such as fetching, parsing, and URL filtering.
In addition to its basic functions, StormCrawler offers integration components like spouts and bolts compatible with Elasticsearch and Apache Solr, along with a ParserBolt that employs Apache Tika to process multiple document types. This adaptability makes it suitable for a variety of organizational requirements, playing a key role in powering the extensive public news dataset of Common Crawl.
The project has garnered industry recognition, with interviews featured on Linux.com and InfoQ, as well as a comparative evaluation against Apache Nutch published on DZone. Academic studies have also documented the use of StormCrawler in initiatives such as large-scale health-related web crawling, creation of Persian language corpora, and development of security information retrieval systems.
To aid users and contributors, the project Wiki maintains an extensive collection of resources including presentations and video materials.