Preservation of Historical Data: Using Warchaeology to Process 20 Years of Harvesting
Presentation outlines how the National Library of Norway has used their open-source tool, Warchaeology, to process 2PB of data, harvested since 2001. The objective is better management and preservation, including to identify collections and groupings of data, parse and sort metadata, identify formats and how these should be processed or converted, deduplicate files, and gather insight about collections generally. It was presented at the IIPC General Assembly and Web Archiving Conference held on April 8-10, 2025 in Oslo, Norway.