Heritrix

Heritrix

Company type	Free software
Industry	Web archiving, web crawling
Founded	January 2004 (2004-01)
Founder	Internet Archive and Nordic national libraries
Headquarters	United States
Products	Web crawler software
Website	webarchive.github.io/heritrix/

Heritrix is a web crawler specifically created for web archiving purposes by the Internet Archive. Developed in Java and distributed under a free software license, its primary interface operates through a web browser, with an optional command-line utility available for initiating crawl operations.

Conceived collaboratively by the Internet Archive alongside Nordic national libraries based on early 2003 requirements, Heritrix saw its inaugural official release in January 2004. Since then, it has undergone ongoing improvements driven by both Internet Archive personnel and external contributors.

For an extended period, Heritrix was not the chief crawler utilized by the Internet Archive for web data collection. Up until 2011, the predominant source of web content was Alexa Internet, whose crawler ia_archiver gathered data primarily for its own use before transferring it to the Archive. The Internet Archive employed Heritrix on a limited basis until 2008, when enhancements were made to scale its performance for extensive crawling tasks. Presently, Heritrix handles the majority of the Archive’s web content acquisition.

Projects using Heritrix

A variety of institutions and national libraries incorporate Heritrix into their web archiving workflows, including:

• Austrian National Library, Web Archiving
• Bibliotheca Alexandrina
• Bibliothèque nationale de France
• British Library
• California Digital Library's Web Archiving Service
• CiteSeerX
• Documenting Internet2
• Internet Memory Foundation
• Library and Archives Canada
• Library of Congress
• National and University Library of Iceland
• National Library of Finland
• National Library of New Zealand
• Royal Library of the Netherlands (Koninklijke Bibliotheek)
• Netarkivet.dk
• National Library of Israel

Arc files

Earlier iterations of Heritrix saved captured web content within Arc files, a format distinct from the similarly named ARC (file format). Since 1996, the Internet Archive has utilized Arc files for storing web archives but now defaults to the more adaptable and rigorously defined WARC file format. Additionally, Heritrix can organize saved data in a directory layout akin to that used by the Wget crawler, employing URLs as filenames.

An Arc file aggregates numerous archived resources into a single container, facilitating easier management compared to handling many individual small files. Each file consists of consecutive URL records, including metadata headers followed by the respective HTTP header and response body.

Tools for processing Arc files

Heritrix offers arcreader, a command-line utility designed to extract contents from Arc files. For instance, retrieving a list of all URLs along with metadata in CDX format can be done by:

arcreader IA-2006062.arc

To extract a particular resource such as hello.html at a specified offset:

arcreader -o 140 -f dump IA-2006062.arc

Other available utilities include:

• Arc processing tools
• WERA (Web ARchive Access)

Command-line tools

Heritrix includes multiple command-line programs that assist with crawl oversight and data extraction:

• htmlextractor – reveals links that Heritrix would extract from a specified URL
• hoppath.pl – reconstructs the navigation path to a URL from completed crawl data
• manifest_bundle.pl – compiles all resources listed in a crawl manifest into compressed or uncompressed tar archives
• cmdline-jmxclient – enables command-line interaction with Heritrix’s control interface
• arcreader – extracts contents from ARC files (as described above)

Further utilities are accessible through the Internet Archive’s warctools project.

See also

• Internet Archive
• National Digital Information Infrastructure and Preservation Program
• Web crawler

References

2. Burner, M., "Crawling towards eternity – building an archive of the World Wide Web," Web Techniques, 1997, vol. 2, no. 5, available at http://www.webtechniques.com/archives/1997/05/burner/
3. Mohr, G., Kimpton, M., Stack, M., Ranitovic, I., "Introduction to Heritrix, an archival quality web crawler," in Proceedings of the 4th International Web Archiving Workshop (IWAW’04), 2004, accessible at http://www.iwaw.net/04/Mohr.pdf
4. Sigurðsson, K., "Incremental crawling with Heritrix," presented at the 5th International Web Archiving Workshop (IWAW’05), 2005, found at http://www.iwaw.net/05/papers/iwaw05-sigurdsson.pdf

Internet Archive Tools

• Heritrix 3 Documentation
• NutchWAX – tool for searching web archive collections
• Wayback (Open source Wayback Machine) – enables browsing and searching of web archives via NutchWAX

Associated Tools

• Arc file format
• Running Heritrix on Windows
• WERA (Web ARchive Access) – software for exploring archives using NutchWAX