title: Reach for the (cloudy) Sky! Exploring cloud-to-cloud archiving of digital records
creators: Kirsten Arnold
abstract: The National Archives (UK) is committed to making transfer of digital records faster, easier and more scalable. Our Transfer Digital Records (TDR) service allows depositors to upload records from a network drive environment. However, an increasing number of users (c86% of UK public record bodies) now use cloud ecosystems, mostly SharePoint but also Google drive. Our current digital transfer process requires users to extract records from the cloud before transfer (through re-upload to the cloud). This is slow, inefficient, error-prone and, most importantly, can disrupt or lose metadata (particularly dates). It also introduces challenges in selecting export formats. A significant volume of records are affected – both current records and legacy data that has been migrated to the cloud. Scalability is essential.
  
  This lighting talk will describe a feasibility study into enabling users to send records directly from depositors’ SharePoint services to the Archive, quickly, securely and without loss of metadata or file integrity. The study engaged with external partners including Microsoft.
  
  Criteria for maintaining integrity were identified:
  •	Transfer user is clearly identified throughout the process
  •	Transfer consignments are identified and tracked
  •	Users can select which records to transfer
  •	The Archive can securely access only selected records
  •	Comprehensive metadata is harvested without disruption
  •	Metadata is in a form suitable for ingest into archival systems
  •	Files are harvested and uploaded to the Archive for ingest
  •	Harvesting triggers AWS step-functions which automatically initiate archival processing
  •	User administration, registration and compliance policies are met
  •	Security and accessibility standards are met
  
  The study identified implementation options including full or minimal integration with a 3rd party app and bespoke development. The talk will outline the options and discuss some outstanding questions which we continue to explore:
  
  •	Research into usability and adoption rates
  •	Research into how depositors organise SharePoint metadata
  •	Understanding how the Archive wishes to consume and process harvested records and metadata, including custom metadata and tags
  •	Scaling constraints on records size or volume
  •	Sustainability of the app approach, including ownership models, costs, maintenance, support, exit strategy
keywords: metadata standards and implementation scaling up