Better Together: Building a Scalable Multi-Crawler Web Harvesting Toolkit
Presentation outline some of the many lessons and best practices the Internet Archive has learned from the challenges, requirements, research, and practical experience from collaborating with other memory institutions for over 25 years to meet the harvesting needs of the preservation community. It was presented at the IIPC General Assembly and Web Archiving Conference held on April 8-10, 2025 in Oslo, Norway.