Politely Downloading Millions of WARC Files Without Burning the Servers Down
Poster sharing the development and release of an open source, cross-platform, dependency-free and user-friendly tool that implements an HTTP client with an easy to configure retry-strategy with exponential backoff and jitter. The introduction of these more polite retry strategies allow users to avoid download errors, and to more quickly download data as the exponential backoff with jitter lets the server more easily handle concurrent requests. It was presented at the IIPC General Assembly and Web Archiving Conference held on April 8-10, 2025 in Oslo, Norway.