Summer Institute on Data Curation:
Digital Preservation& Standards
Jerome McDonoughAsst. Professor, GSLIS
June 4, 2008
I love standards. There are so many of them to choose from.
Standards & Sustainability
Disclosure: Are complete specifications available? For free?
Adoption: To what extent is the standard already used?
Documentation: Is the specification clear and straightforward? Are there additional resources to assist in understanding the standard?
Standards & Sustainability
External Dependencies: To what extent does use of the standard rely on particular hardware or software? On other standards? On other non-standards?
Impact of Patents: If patents cover some or all of the standard, are licensing issues likely to complicate use of the standard?
Technological Protection Measures: Does the standard rely on technological protection measures which will inhibit your ability to preserve data?
Tip of the hat to Library of Congress Sustainability Of Digital Formats Sitehttp://www.digitalpreservation.gov/formats/sustain/sustain.shtml
Part I:How to Operate an
Archive
Open Archival Information System
Reference ModelDeveloped by the Consultative Committee For Space Data Systems
Adopted as ISO 14721:2003Available at http://public.ccsds.org/publications/archive/650x0b1.pdf
Provides definitions of components of an archive, their relationship to each other, a set of mandatory responsibilities for an archive, and both functional and data models.
OAIS Reference Model:Mandatory
Responsibilities Negotiate for an accept information from producers Obtain sufficient control of information to ensure long-
term preservation (including necessary IP permissions and authority to migrate)
Determine which communities should be the Designated Communities and should be able to understand the information provided
Ensure that the information to be preserved is independently understandable to the designated community (i.e., they can understand it without the assistance of experts who created it ).
Follow documented policies and procedures ensuring information is preserved against reasonable contingencies
Make the information available to the designated community
OAIS Functional Model
OAIS Functional Model: Ingest
OAIS Functional Model: Archival Storage
OAIS Functional Model:Data Management
OAIS Functional Model: Access
OAIS Functional Model:Preservation Planning
OAIS Functional Model: Administration
OAIS Data Model
Part II:How to Create Content
for an Archive
Archival Content
A syllogism to ponder:No digital media can be read without a hardware device designed to read the media format.
It is exceedingly rare for a hardware device intended to read a specific digital media format to be manufactured for more than 30 years, and many have had shorter lifespans.
Therefore, if your content is not device independent, it is not really archival.
Archival Content: Text
Some Issues to Consider When Examining Text StandardsTechnical aspects of character encoding
Character Repertoire (Script & Language Support)
Line Break Handling & Line OrientationIndexingFormattingOther processing
Archival Content: Text
A Standard for CharactersUnicode 5.1 - ISO/IEC 10646
Two variable length encodings (UTF-8, UTF-16) and a fixed length encoding (UTF-32). In UTF-8, byte order is not an issue. In UTF-16 and UTF-32, big-endian and little-endian encodings are supported.
Over 100K characters, supporting 75 different scripts and many additional symbols and diacritics, with room for expansion to 1,114,112 characters.
Support for a variety of line breaking mechanismsSupport for different text directionality, including algorithms specifying the appropriate handling of text of mixed directionality
Archival Content: Text
A Standard for Syntax XML (World Wide Web Consortium)
Standards for Semantics Chemical Markup Language, Chemical Industry Data Exchange
Astronomical Markup Language, Astronomical Dataset Markup Language, Astronomical Instrument Markup Language
Earth Science Markup Language, Geography Markup Language, NetCDF Markup Language, ArcGIS Markup Language
MathML Etc., etc., etc….
Archival Content: Images
Some Issues to Consider When Examining Image StandardsColor DepthColor SpaceColor ManagementImage Resolution ScalabilityCompression
Archival Content: Images
Tagged Image File Format (TIFF) 6.0 -- 1 to 64-bit color depth, supports grayscale, RGB, YCbCr, CMYK and CIELab color spaces, supports embedded ICC color profiles, raster format, supports uncompressed as well as lossless and lossy DCT-based compression
JPEG 2000 (ISO/IEC 15444) -- 1-48 bits per channel with multiple channels (including alpha & transparency), supports wide array of color spaces with sRGB and sYCC as defaults, supports ICC color profiles, raster format, supports uncompressed as well as lossless and lossy wavelet based compression
Scalable Vector Graphics 1.2 -- uses sRGB color spaces, supports ICC Color Profiles, vector format
Archival Content: Audio/Video
Some Issues to Consider when Examining Audio/Video StandardsAudio sampling rateAudio bit depthVideo frame rateVideo color space/depthCompression
Good News: Audio/Video is a bit more standardized than text/image world
Bad News: Lossless digital audio is rare; lossless digital video is almost nonexistent.
Archival Content:Audio/Video
Broadcast WAVE Audio (EBU Standard N22 - 1997)
For video, picture is less clear. Proprietary solutions dominate market. Many of these (e.g., QuickTime, WMV) do support lossless image frame and audio data. MXF, a SMPTE standard, is gaining some traction in digital library circles (and the movie industry)
Archival Content: Data
Some disciplinary de facto standards (e.g., Chemical Markup Language). Cover Pages (http://xml.coverpages.org) is a good source for information on many of the major ones.
No single standard for general use for data encoding, although many contenders
Archival Content: Data
Binary Format Description Language (BDFL) -- XML language based on the Extensible Scientific Interchange Language (XSIL) that supports documentation of binary and ASCII data
eXtensible Data Format (XDF) -- scientific data format supporting hierarchical data structures, N-dimensional arrays, scalar and vector fields, user-defined coordinate systems
Archival Content: Data
Data Format Description Language (DFDL) -- A language for describing the structure or binary and character encoded data to expose their structure, format and metadata so that machine processes can work upon them.
Data Documentation Initiative (DDI) -- An effort by the ICPSR at Univ. of Michigan to develop an XML format for documenting social science data sets. XML files can be used to produce either bibliographic descriptions of data sets or SAS/SPSS/STATA data definition statements.
Archival Content: Data
Hierarchical Data Format (HDF5) -- General purpose file format (with supporting software library) for storing scientific data, developed by NCSA. Uses two fundamental structures, groups and data sets, where a data set is an N-dimensional array of data elements with metadata.
Archival Content: Paper
ANSI/NISO Z39.48-1992, Permanence of Paper for Publications and Documents in Libraries and Archives
ISO 9706-1994, Information and documentation -- Paper for documents -- Requirements for permanence
Part III:How to Create Metadata
for an Archive
Metadata: Identifiers
Persistence is important, but…Clarity on what is being identified may be more important (or, why an OpenURL is not a call number).
Standards proliferate in this space; choice of any identifier may depend on:Social concerns (for whom am I identifying something?)
Identifier/address resolution (how do I find a copy/item using this identifier?)
Metadata: Structural
Metadata intended to identify the components of an object and their relationship to each other in order to support the object’s navigation and use
Metadata Encoding & Transmission Standard (METS)
MPEG-21 Digital Item Declaration Language
XML Formatted Data Units (XFDU)OAI-ORE
Metadata: Provenance
Metadata documenting the origins and life-cycle of a digital object
PREMIS Data Dictionary for Preservation Metadata 2.0Joint project of OCLC & RLGDefines metadata element set that “supports the viability, renderability, understandability, authenticity and integrity of digital objects in a preservation context.”
Metadata: Provenance
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
The PREMIS Data Model
Metadata: Provenance
PREMIS Object Metadata:IdentifierCategoryPreservation LevelSignificant PropertiesCharacteristics (fixity, size, format, etc.)Original NameStorageEnvironmentSignatureRelationships to other Objects, Events, Rights
Metadata: Provenance
PREMIS Event MetadataIdentifierTypeDate & TimeDetailsOutcomeRelationship to Agents and Objects
Metadata: Provenance
PREMIS Agent MetadataIdentifierNameType
Metadata: Provenance
PREMIS Rights MetadataRights StatementRights BasisCopyright InformationLicense InformationStatute InformationRights GrantedRelationship to Objects and Agents
Metadata: Administrative
Technical MetadataZ39.87 and MIXTechnical Metadata for Text (TextMD)AES-X098 Administrative Metadata for Audio Objects
SMPTE RP210.10-2007 Metadata Dictionary
Rights MetadataStandards, yes. That you want to use, no.
Metadata: Descriptive
Issues to consider:Nature of object to be describedReal purpose(s) of descriptionCommunity(ies) that will utilize description
Supporting standards of descriptive practice and controlled vocabularies
Metadata: Descriptive
Library/Archives/Museums/EducatorsMARC, MODS, Dublin CoreEADVRA Core, CDWAIEEE LOM
Data RepositoriesData Documentation InitiativeContent Standard for Digital Geospatial Metadata
Darwin CoreAccess to Biological Collection Data (ABCD)
How to Evaluate an Archive
Evaluating Archives
Trustworthy Repositories Audit & Certification (TRAC) Criteria & Checklisthttp://www.crl.edu/content.asp?l1=13&l2=58&l3=162&l4=91
Digital Repository Audit Method Based on Risk Assessment (DRAMBORA)http://www.repositoryaudit.eu/
Exercise: URLs
Imageshttp://people.lis.uiuc.edu/~jmcdonou/Bryce.tif
http://people.lis.uiuc.edu/~jmcdonou/Bryce.jp2
Exercise: URLs
METS Schema, Documentation, Namespace http://www.loc.gov/standards/mets/mets.xsd http://www.loc.gov/standards/mets/docs/mets.v1-7.html http://www.loc.gov/METS/
PREMIS Schema, Documentation, Namespace http://www.loc.gov/standards/premis/v1/PREMIS-v1-1.xsd
http://www.loc.gov/standards/premis/ http://www.loc.gov/standards/premis/v1
MIX Schema, Documentation, Namespace http://www.loc.gov/standards/mix/mix20/mix20.xsd http://www.niso.org/kst/reports/standards?step=2&gid=
None&project_key=b897b0cf3e2ee526252d9f830207b3cc9f3b6c2c
http://www.loc.gov/mix/v20