The Provenance of Web Archives
The Provenance of Web Archives
Over the last few years, it’s been wonderful to see more and more researchers taking an interest in web archives. Perhaps we are even teetering into the mainstream when a publication like Forbes carries an article digging into the gory details of how we should document our crawls in How Much Of The Internet Does The Wayback Machine Really Archive?

Even before the data-mining BUDDAH project raised these issues, we’d spent a long time thinking about this, and we’ve tried to our best to capture as much of our own crawl context as we can. We don’t just store the WARC request and response records (which themselves are much better at storing crawl context than the older ARC format), we also store:

The list of links that the crawler found when it analysed each resource (this is a standard Heritrix3 feature).
The full crawl log, which records DNS results and other situations that may not be reflected in the WARCs.
The crawler configuration, including seed lists, scope rules, exclusions etc.
The versions of the software we used (in WARC Info records and in the PREMIS/METS packaging).
Rendered versions of original seeds and home pages, as PNG and as HTML, and associated metadata.