Digital Preservation Community Resources

• Get Started
  • Save Digital Stuff Right Now
  • Preserve Your Own Stuff
  • Become Part Of The Digital Preservation Community
• Real Data and Requirements
  • Test Corpora
    • Multi-format Corpora
    • Format-specific Corpora
    • Building Corpora
    • Sourcing test files from web archives
  • Tools
  • Building Workflows
  • Understanding Formats
    • Improving Identification
    • Improving Characterisation/Metadata Extraction

Get Started

Save Digital Stuff Right Now

Spotted digital data at risk, but don’t know who can save it?

• Save web pages via:
  • Internet Archive Nominations
  • archive.is
  • perma.cc
  • webcitation.org
  • UK Web Archive Site Nomination
• Alert the Archive Team, and help them save digital stuff

Preserve Your Own Stuff

• The Getting Started chapter of the Digital Preservation Handbook is a great place to start.
• For material that describes the broader issues, you can refer to Digital Preservation on Wikipedia, and consider contributing to the Digital Preservation Wikipedia Project.
• Build your roadmap, guided by the NDSA Levels of Digital Preservation
• Use the Digital Preservation Business Case Toolkit to help get funding.
• Understanding your costs can help to plan your preservation work more effectively. The Curation Costs Exchange allows you to compare your costing data with that of many other organisations.
• Learn about Preserving digital Objects with Restricted Resources
• Share your challenges and solutions with the community, and view the resulting record of digital preservation practice.
• Learn how to Authenticate, Manage, and Preserve Video – WITNESS trains activists to archive and preserve their video so that human rights abuses cannot be denied or forgotten over time.
• Engage Stakeholders:
  • Visual examples of digital preservation challenges, such as graphic corruption, can be incredibly useful in communicating the digital preservation message. That’s why we built the Atlas of Digital Damages Gallery and website. Add an image of a digital preservation challenge, failed rendering, encoding damage, corrupt data, or visual evidence documenting to the Atlas of Digital Damages.
  • Use the POWRR One Pagers to educate stakeholders about the issues.
  • Working with your IT department (some responses arising from this question on twitter:
    • Backup versus preservation: “I had useful discussions about ‘for the long-term’ and what issues that might throw up as a starting point.”
    • “often it comes down to language - avoid the word archiving…means 1 thing to IT & another DP professionals”
    • Ten IT skills you need to have to work with digital preservation, by Dave Thompson
    • “Am going to go all Bruce Lee ‘adjust to the object’. What drives IT? Delivery, efficiency, user needs etc.”

Become Part Of The Digital Preservation Community

Advance digital preservation by pooling our experience, sharing our war stories and finding the answers to the big questions.

• Q&A:
  • Ask and answer digital preservation questions
  • We tried to run a Digital Presevation Stack Exchange, but it didn’t work out. The content is available here
• Forums
  • Discussion forums and active blogs provide the opportunity to share informal advice and war stories, get recommendations and discuss the finer points of digital preservation. By sharing both your intentions for digital preservation work and your results, you can ensure your work benefits from a wealth of community experience.
  • Discuss preservation issues on the Digital Curation forum
  • Share war stories on OPF blogs
  • Mastodon - Join these federations with a digital preservation or general GLAM focus:
    • https://digipres.club
    • https://glammr.us
    • https://ausglam.space
    • https://code4lib.social
  • Twitter - Use these lists to find people to follow:
    • https://twitter.com/DavidUnderdown9/lists/digipres1
    • https://twitter.com/An_Old_Hand/lists/digi-pres-rdm
    • https://twitter.com/digipresnews/followers
    • https://twitter.com/NKrabben/lists
  • r/DataHaorder – “We are digital librarians.”
  • r/Archiveteam – “Archive Team is a loose collective of rogue archivists, programmers, writers and loudmouths dedicated to saving our digital heritage.”
  • Join the Digital POWRR Slack
• Face-to-Face communities/support groups:
  • Digital Cultural Heritage DC
  • Mid Michigan Digital Practitioners
  • XFR Collective: NYC-based media preservation membership organization preserving at-risk and obsolete audiovisual artwork by providing low-cost migration services
• Collaborations (inc. groups that build things together):
  • Association of Moving Image Archivists (AMIA) Open Source Committee on GitHub
• Conferences:
  • PASIG
  • iPres
• Membership organisations:
  • DPC (International)
  • OPF (International)
  • NCDD (Netherlands)
  • NDSA (USA)
  • NESTOR (Germany)

Real Data and Requirements

Real data, real challenges and real requirements make your and others digital preservation developments far more useful and effective.

• Test tools using publicly-available test corpora (see below)
• Add your files to the OPF Format Corpus
• Contribute to the DP Requirements and Solutions wiki
• In order to improve our digital preservation capability, we need to be able to analyse and evaluate our work in an effective manner. For example, we need to be able to compare and contrast tools and approaches, and we need to see how changes over time affect performance. Practising what we preach in this field means sharing our data about digital preservation. Share your digital preservation data using the Linked Data Simple Storage Specification. Dave Tarrant explains why this is a good idea.
• See also Where are the Born-Digital Archives Test Data Sets?

Test Corpora

To improve our digital preservation tools, we need to be able to test them and evaluate of their performance. Publicly available sample files make this much easier. Tool developers can use them to test their work, discover bugs, and hone their tools ready for others to use. A test corpus can contain real digital objects from a collection, or be created specifically for exhibiting certain characteristics for testing purposes. Real data, particularly with examples of broken, badly formed or corrupted files can be particularly useful.

Note that OPF also has it’s own corpora page.

Multi-format Corpora

• The OPF Format Corpus
• The iPres System Showcase Test Suite
• The Encyclopedia of Graphics File Formats Companion CD-ROM contains lots of test files for image formats:
  • EGFF ISO Image
• EDRM Data Set Files
  • EDRM File Formats Data Set 1.0.1
  • EDRM Internationalization Data Set
• Digital Corpora’s corpora
  • Including govdocs1, also described here
  • OPF have also created a by-format subset of govdocs1, which you can read more about here
• digicam corpus contains a corpus of Digital Camera files collected by Tyler Thorsted
• The Skeleton Test Suite builds test files from PRONOM binary and container signatures. These can be used to test DROID and other (compatible) identification tools.
• Fine Free File Test Suite set up for Fedora testing
• JHOVE’s test files
• JHOVE2’s test files
• The disktype test files
• The Metadata Working Group specifications and embedded image metadata test corpus
• Apache Tika issue about setting up a nightly test corpus, see also tika-parsers/src/test/resources/test-documents
• The Chemical MIME Home Page
• Online-convert.com example files (use this link to browse the folder structure)
• RDSS Archivematica Test Data Corpus A collection of research dataset files used for testing Archivematica integration and functionality in the JISC Research Data Shared Service (RDSS).
• Archivematica Sample Data Includes OPF format corpus, as well as other test material
• ExifTool test files
• PREFORMA Ground Truth Classes Instructions how to reproduce validation-failing files for Matroska, FFV1, LPCM, TIFF, and PDF formats
• “Small” Collection of “the smallest possible syntactically valid files in different programming/scripting/markup languages.”
• MediaArea-RegressionTestingFiles Public regression testing files for MediaArea. Contains AVI, FLV, MPEG Audio, MOV, MPEG-4, MPEG-PS, and Matroska files.
• TechSlides sample files for web development Sample files for various image formats, video files, data structures, fonts, and web development files.

Format-specific Corpora

• PDF:
  • Adobe Acrobat Engineering site has lots of useful test documents.
  • Isartor PDF/A Test Suite
  • veraPDF Corpus for PDF/A
  • Synthetic PDF Testset for File Format Validation Test set for well formedness validation in JHOVE - see associated paper
• ePub:
  • The IDPF ePub test suite
  • KBNLresearch/epubPolicyTests - Some #epub samples with encryption, DTBook content and foreign resources, with corresponding #epubcheck output.
• TIFF:
  • The libtiff TIFF Test Images
• JP2:
  • OPF JP2k test corpus
  • NITF version 2.1 JPEG 2000 Sample Imagery
  • JPEG 2000 Part 4 Conformance Test Files (v.1.5 with earlier versions also available in the archive history)
• WARC/ARC:
  • Internet Archive’s Example ARC and WARC files
• SIARD:
  • SIARD test files

Building Corpora

If the existing corpora aren’t cutting it, perhaps you can contribute to the OPF Format Corpus (hosted on GitHub). There’s a guide here on how to contribute or you can contact OPF for help on how to get involved.

Sourcing test files from web archives

Web archives can provide a useful source of files of particular formats. For example, search via the UKWA interface.

Tools

Software tools give us the means the interrogate, manipulate, understand and ultimately preserve our digital data.

• Find tools to solve your challenges with the POWRR Tools Grid
• Contribute your experiences of using tools to the COPTR wiki
  • The Community Owned digital Preservation Tool Registry, COPTR has unified five isolated tool registries. It provides an easy-to-edit wiki interface where we can share our knowledge about, and experiences with, tools used for digital preservation purposes. If you create a new digipres tool, please add it to COPTR, but before you create a new tool, please use COPTR to find similar ones and see if you might be able to extend or improve some of the existing tools.
• Contributing to the development and improvement of tools is easy, even if you’re not technical. Check out this guide to making small documentation edits, or raising issues on Github.
• Digital preservation needs high-quality tools, but to get high-quality tools, we need rich and varied test corpora to stretch those tools to their limits.

Building Workflows

Resources to help build up preservation workflows, e.g. templates for how to use command-line tools, and how to chain things together.

• ffmprovisr ‘Making FFmpeg Easier’ (example of how to use ffmpeg to perform specific tasks)
• AMIA Open Source: List of open workflows for A/V resources

Understanding Formats

We need to understand the file formats of the resources we care for, and the software they depend on.

• Search for formats
• Add formats to the File Formats Wiki
• Understand file format risks (e.g. JP2)
  • If you have good examples of digital resources and their risks, please consider adding them to a test corpus.
• Enhance file format identification
• Identify formats now with FIDOO, Siegfried or Interject

Improving Identification

Identifying file formats is the bread and butter of digital preservation characterisation and assessment. Identification tool coverage and accuracy could be much better, and this primarily comes down to the signatures, or file format “magic”, used to identify each format. You can help contribute and make our identification tools more effective here:

• Submit a file format signature/patch to Apache Tika, like this one.
• Contribute a file format signature to DROID. See this guide and this official guide.
• Contribute a file format signature to FILE. See this guide.

If you want to start to put this into practice you can identify file formats right now (with no installation or setup) using FIDOO or alternatively check out stand alone file format identification tools.

Improving Characterisation/Metadata Extraction

Deep file characterisation enables validation, identification of preservation risks and extraction of metadata. In developing a new characterisation capability, begin with thorough research to identify existing code to re-use or build on, develop a focused command line tool, then consider turning it into a JHOVE module.

• Develop a new file characterisation capability and turn it into a JHOVE module, or an Apache Tika module.