Apache Nutch
| Company type | Open source software |
|---|---|
| Industry | Software development |
| Founded | June 2003 |
| Founder | Doug Cutting, Mike Cafarella |
Area served | Worldwide |
| Products | Web crawler software |
| Owner | Apache Software Foundation |
| Website | nutch |
Apache Nutch represents a highly adaptable and scalable open source web crawler software initiative.
Features
Nutch is fully implemented in Java, utilizing data formats that remain independent of any specific programming language. Its design is modular, allowing developers to build plug-ins for tasks such as media-type parsing, data acquisition, querying, and clustering.
The fetcher component—often referred to as the "robot" or "web crawler"—was custom-developed exclusively for this platform.
History
The inception of Nutch traces back to Doug Cutting, the creator behind Lucene and Hadoop, alongside co-founder Mike Cafarella.
By June 2003, a demonstration system capable of handling 100 million pages was finalized. To facilitate distributed processing across multiple machines for crawling and indexing, Nutch integrated the MapReduce paradigm along with a distributed file system, which subsequently evolved into the standalone Hadoop project.
Nutch joined the Apache Incubator in January 2005, became a subproject under Lucene by June 2005, and achieved status as an independent top-level project within the Apache Software Foundation in April 2010.
In February 2014, the Common Crawl initiative adopted Nutch to support its open large-scale web crawling operations.
Although initially designed with the ambition to develop a global-scale web search engine, this objective is no longer actively pursued.
Release history
| 1.x
Branch |
2.x
Branch |
Release date | Description |
|---|---|---|---|
| 1.1 | 2010-06-06 | Significant updates to dependencies including Hadoop, Solr, and Tika; bug resolutions; and enhanced performance for Fetcher2. | |
| 1.2 | 2010-10-24 | Introduced parse-html parser option, configurable per-field indexing capabilities, timing metrics for Tool classes, parser timeout settings, plus various bug fixes. | |
| 1.3 | 2011-06-07 | Enhanced RSS parsing features; deeper integration with Apache Tika; support for external parsers; improved language identification; and reduction in source code size. | |
| 1.4 | 2011-11-26 | Expanded MIME type support within parsers; adjustable Fetcher queue depth; speed optimizations; and HTTP authentication support added for Solr indexing. | |
| 1.5 | 2012-06-07 | Updated Tika and Hadoop dependencies; improved LinkRank and WebGraph functionalities; added new plugins for blacklisting, filtering, and parsing tasks. | |
| 2.0 | 2012-07-07 | Emphasized large-scale crawling capabilities with storage abstraction via Apache Gora supporting major big data stores such as Accumulo, Cassandra, HBase, and HDFS. | |
| 1.5.1 | 2012-07-10 | Maintenance update for the widely used 1.5.x series. | |
| 2.1 | 2012-10-05 | Streamlined distribution process with enhanced Solr configuration properties; upgraded Gora dependencies; introduced ElasticSearch indexing option. | |
| 1.6 | 2012-12-06 | Resolved over twenty bugs; added HostNormalizer feature; enabled dynamic fetchInterval based on MIME types; improved Indexer API; upgraded Tika and Automaton libraries. | |
| 2.2 | 2013-06-08 | Fixed more than thirty bugs; integrated Crawler-Commons library for robots.txt parsing; updated Hadoop, Gora, Tika, and Automaton dependencies. | |
| 1.7 | 2013-06-24 | Launched a new pluggable indexing framework supporting both Solr and ElasticSearch; delegated robots.txt parsing responsibilities to Crawler-Commons; included key library upgrades. | |
| 2.2.1 | 2013-07-02 | Bug fixes addressing ByteBuffer to String conversion issues; updated Hadoop and Tika libraries. | |
| 1.8 | 2014-03-17 | Upgraded Crawler Commons and Tika libraries; implemented over thirty bug fixes alongside multiple enhancements. | |
| 2.3 | 2015-01-22 | Released a self-contained web application based on Apache Wicket; deprecated SQL backend support for Gora. | |
| 1.10 | 2015-05-06 | Tika upgrade included; addressed more than forty-six bugs and introduced thirty-seven improvements plus new features. | |
| 1.11 | 2015-12-07 | Hadoop upgraded to version 2.x along with Tika updates; resolved over thirty-two bugs with additional enhancements and features. | |
| 2.3.1 | 2016-01-21 | Bug fix update resolving approximately forty issues. |
Scalability
IBM Research assessed the capabilities of Nutch/Lucene within its Commercial Scale Out (CSO) initiative. Their results demonstrated that a scale-out architecture, such as Nutch/Lucene operating on a blade server cluster, can deliver performance beyond what any scale-up machine like the POWER5 can provide.
The ClueWeb09 collection, utilized in the Text Retrieval Conference (TREC), was gathered using Nutch, achieving an average crawling rate of 755.31 documents per second.
Related projects
- Hadoop – A Java-based framework designed to support distributed computing across extensive clusters.
Search engines developed with Nutch
- Common Crawl – Offers publicly accessible web-wide crawl data, initially launched with Nutch in 2014.
- Creative Commons Search – Employed Nutch between 2004 and 2006.
- DiscoverEd – A prototype search engine for open educational resources created by Creative Commons.
- Krugle – Utilizes Nutch to index web content focused on code and technical information.
- mozDex (no longer active)
- Wikia Search – Introduced in 2008 and discontinued in 2009.