pdptv-austin-ischoolbesser.tsoa.nyu.edu/howard/Talks/11pdptv-austin-ischool.pdf · Examined Potential Points of Metadata Capture Besser, PDPTV, Austin, 3/10/11 Examined Potential

3/9/11

1

Building sustainable Digital Libraries: Lessons from NDIIPP's Preserving Digital

Public Television project Howard Besser

New York University http://dlib.nyu.edu/pdptv/

Besser, PDPTV, Austin, 3/10/11

NDIIPP Preserving Digital Public Television Project-

•  Background & Goals •  Relevance to other domains •  Pushing Metadata Gathering Upstream •  Other sub-projects

–  Examining user needs for selection/quality –  Intellectual Property Issues –  Repository Design –  Sustainability Issues

•  Value of Partnerships •  Creation of American Archive


Background & Goals:

NYU/Public Television Project •  2004-2010 •  $6 million project -- 50% from LC/NDIIPP •  Marry asset management to preservation •  Preserve a broad set of elements (including ancillary material) •  Life-cycle mgmt (add metadata as soon as a clip comes in) •  Establish a community of stakeholders, working together for

preservation (stations, university, librarians, journalists, historians, producers, scholars, …)

•  Build an OAIS Repository •  Explore appropriate file formats, wrappers, METS extensions •  Develop sustainable business model


Project Partners •  Thirteen/WNET & WGBH – Content and production expertise

–  The two largest television stations in the PBS system –  Together produce largest percentage of national programs –  Both have preservation Archives

•  Public Broadcasting Service – More content and network design –  Distributes most of the national programming –  Determines and keeps ‘broadcast’ versions

•  New York University – Facilitation and Resources –  Leadership in designing digital libraries –  Experience in process for setting standards –  Has new Masters Program in Moving Image Archives


Public Television was analog for a long time


PBS Remote Tape Storage Facility


3/9/11

2

Computer-printed index of�1955-69 WNET Holdings at LC


By 2004, Public Television had already become Digital for Production & Distribution

•  But not for Archiving •  and some things still need to be streamlined


Programs were shot, assembled and edited digitally. Completed programs were packaged as digital files,

often with many different elements.

Typical AVID Editing Suite

The same huge video files are digitized over and over again for different uses.

A lot of video is not used for broadcast, but goes to the internet, to DVDs and to other media.


Local broadcasting was nearly all via digital systems

Thirteen Master Control manages one high definition, two analog and three digital over-the-air broadcast channels.


Broadcast playback was from a digital server - no more tapes or tape machines

All Thirteen broadcast programs are played from this server.

It has to talk to the broadcast automation system, the satellite system, and other in-house and external networks.

It is the size of a window air conditioner and holds 1700 hours of material.

Besser, PDPTV, Austin, 3/10/11 Besser, PDPTV, Austin, 3/10/11

Challenges of Digital brought the partners together

•  Formats & Versions issues (including variant forms of content, HD/SD quality, …)

•  Agreements on technical standards for handing off files from one organization to another

•  Repository design for long-term sustainability

3/9/11

3


Project activities include •  Completing an inventory of at-risk materials to better quantify

our holdings and prepare for selection

•  Reviewing best practices and most up-to-date developments in the field of video archiving

•  Conducting facilitated discussions on key topics to guide setting standards and policies

•  Establishing an Advisory Committee to assist with selection criteria

•  Ingesting sample materials and testing the repository

•  Presenting regular reports to public broadcasting and moving image archive community for ongoing feedback


Project Focus Areas

•  Appraisal and Selection – developing criteria and standards for what to preserve and by whom

•  File Formats and Packages – determining the best formats for our various uses, plus testing the suitability of file “packaging” for long term preservation

•  Metadata and Related Topics – specifying technical, descriptive and rights information

•  Repository Design – technical architecture, administrative policies and potential business models

•  Sharing Our Findings – Keeping the public broadcasting community involved and informed all along the way


By 2007, the Collaboration had already show success

•  Helped produce standards that allow files to flow digitally from producer to PBS to stations

•  Studied various metadata schemes to zero in on what’s needed for preservation

•  Inventoried and located programs in non-regular locations

•  “Wrapper Roundtable” & aftermath (METS, MXF, AAF)

•  PBS & LC beginning to collaborate •  The “American Archive”

Pushing Metadata Gathering Upstream: The Problem

TRADITIONALLY… •  Very little metadata required for

preservation accompanies an object to a repository.

•  Archives, libraries and other repositories must create (or re-create) most of the necessary metadata.

•  This requires many manual hours, and significant resources - both time and money.

IN THE DIGITAL WORLD… •  This doesn’t scale up. Repositories

will be unable to continue in this manner, as more metadata than ever is required.


But much of the necessary metadata has already been gathered during production

•  For each element/clip, production team usually notes source, date, place, people, and other descriptive info

•  But this is treated as internal information, and often various parts of the info are distributed among the personal notebooks of different production assistants

•  There is seldom a central location for this info, and the info is seldom turned over to the archive (which later tries to recreate much of it)


When the Archive tries to re-create this info, it is seldom successful

Producers know much more about the content of their productions than the archivists do. Archivists wanting accurate info must go back to the production staff (often years later) to start brainstoriming over the info

“Once the (television) program is finished, it is passed on to the archive or library for safe keeping. Librarians will catalog and classify the content, possibly using a proxy copy, and enter the resulting informative metadata in their database so they can retrieve it in the future. However, rarely if ever is the metadata from the rest of the process passed onto them, except, perhaps, for the title, tape number, and basic technical information about recording formats. It has to be re-created, with all the associated risk of errors and lack of accuracy--not to mention the work and time involved.” - Cox, Tadic, and Mulder, Descriptive Metadata for Television (2006)


3/9/11

4

Similar issues w/other content types--E-Journals

•  “The necessary or additional metadata cannot be effectively and satisfactorily produced either as an afterthought post-production process on the publisher’s side or as a pre-ingest conversion activity at the archive’s end. Approaching e-archiving in this fashion leads to distribution delays and a more complex production and distribution scenario, with all the accompanying potential to introduce production delays and errors.”

- Yale University, YEA: The Yale University Archive, One Year of Progress, 2002


We need to find ways to push metadata access upstream

•  Digital requires even more metadata than Analog –  As the workflow becomes file-based, the need for robust and

accurate metadata will become critical. File relationships, video codecs, bit rates, and rights information must be explicit, accurate, and immediately accessible. This will require a much deeper level of metadata than is currently captured in tape-based archives.

–  We can’t continue to supply this metadata at ingest; that won’t scale •  Obtaining the necessary metadata at the end of production and

broadcast life cycle is not feasible. Metadata will need to be systematically gathered during the production lifecycle and submitted with the programs to the preservation repository.



Examined Potential Points of Metadata Capture


Examined Potential Points for Metadata Capture

•  Much of the necessary metadata for preservation is already generated by the production unit, but discarded after their internal use. This needs to be captured throughout the workflow.

•  “Those in the production unit are the creators and have first hand knowledge of who, what, where, when, and why the content was created.” -- Mary Ide and Leah Weisse, WGBH Archivists.

Proposed Solutions…?

•  Preservation becoming a shared responsibility between content creators, distributors, curators, and preservationists.

•  Partnerships are needed to come to unified solutions.

•  Preservationists seek reliable metadata back upstream in the production workflow...


Workflow in Production Process-

•  Site Visits to productions •  Interview Production staff •  Diagrams of Workflow-


3/9/11

5


Site Visits

This report is based on workflow studies at public television stations, between June and August, 2006 by NDIIPP Research Assistants Caroline Rubens, Paula Felix-Dider, and Kara Van Malssen. Workflow report completed in September 2006.

Additional insight was gained through metadata studies conducted by Mary Ide and Leah Weisse, Archivists at WGBH.


Site Visits

•  WGBH, Boston - June 19-20, 2006 –  Interviewees included Archive and Media Library staff,

members of Frontline, NOVA, and American Experience production units, and legal dept staff


• WNET/Thirteen, New York - July 18 and August 2, 2006 – Interviewees included the Archivist, members of Broadway: The American Musical production unit, Broadcast Operations, and Broadcast Technology staff.

• WNET/Thirteen, Washington DC - August 15, 2006 – Religion and Ethics production unit staff interviewed.

Site Visits Public Television Workflow Basically similar to workflows in other fields…


For digital preservation, this shouldn’t be the only place for metadata in the preservation workflow! This is far too late in the cycle!

+


METADATA

METADATA

It also needs to be here!


3/9/11

6

WorldFocus •  Nightly news program begun Oct 2008 •  We began working with Workflows six months before

program began •  Had ability to engineer metadata gathering into the

creation/production process


Other Major Accomplishments-

•  PB Core 2.0 •  Appraisal/Selection/User Needs •  Intellectual Property •  Repository Design •  Sustainability



Helped advance PBCore •  2+ year collaborative effort •  48 elements based on Dublin Core

–  Intellectual content of a media asset or resource -- 13 elements

–  IP-creation, creators, and usage limitations of a media asset or resource -- 7 elements

–  Instantiation (in either digital and/or analog) -- 28 elements

•  http://www.utah.edu/cpbmetadata/

Examining user needs for selection/quality

•  Studies of Users and Potential Users – K-12 and Higher Ed instructors – Journalists – Historians and other researchers – Producers of newer works

•  What types of works do they want, and what quality is needed

•  Appraisal & Selection Report Besser, PDPTV, Austin, 3/10/11

Intellectual Property Report

•  “Intellectual Property and Copyright Issues Relating to the Preservation and Future Accessibility of Digital Public Television Programs” released April 2010


Intellectual Property Report (1/2)

•  What is required to keep digital content usable and accessible, particularly in regard to intellectual property issues –  documenting the experiences of the PDPTV team in building the

prototype repository –  examining production and distribution contracts and documents;

conducting IP audits on target programs –  reviewing current research and commentary on relevant technology

and copyright legislation –  monitoring parallel efforts by our colleagues in various fields, such

as publishing, higher education, information technology, and broadcast television in the US and around the world


3/9/11

7

Intellectual Property Report (2/2)

•  Identifies the context of rights agreements, procedures and economics by which public television programs are produced

•  Provides a detailed explanation of the many types of rights and permissions that can exist for any particular program

•  Discusses the technology sections of the copyright law and their impact on digital preservation

•  Looks at current efforts to amend the laws •  Presents case studies of two different public television series

that had to address major copyright issues before they could be reissued for re-broadcast and non-broadcast distribution


Repository Design

•  Design and develop a preservation repository for born-digital public television content –  Design of preservation environment, technologies

used to support preservation functions –  creation of AIPs for managing complex video files, –  identification of standards needed for various sub

-parts (METS, PBCore, PREMIS), •  (Joe Pawletko)


NYU Preservation Repository

•  Three reference projects: – Afghanistan Digital Library – Hemispheric Institute Digital Video Library – Preserving Digital Public Television

•  Built around DSpace •  METS, MODS, SRW/U, NOID,

versioning …


PTV Metadata and Content

•  PBCore •  MXF wrapped 50Mbps MPEG2 •  AS-PBS MXF wrapped 8Mbps broadcast •  Derivatives - iTunes, Real, YouTube, … •  Transcripts, closed captions •  Promotional material •  …


Item versioning


Components


3/9/11

8

Dspace - SRB

•  We are dealing with big files: – 1 hour at 50Mbps is about 25GB

•  Moving these files around is hard … and slow

•  DSpace doesn’t like files this big •  SRB - Storage Resource Broker


DSpace - SRB


Sustainability

•  Examine issues of long-term content accessibility and recommend methods for sustaining digital preservation of public television materials

•  Look at technical, operational, & economic requirements for sustainable preservation of file-based television production (Kara van Malssen)


6 Strategies: Our Situation

•  Large, diverse, fragmented system •  No bosses/administrators over entire

system


6 Strategies: Our Sustainability Report

•  Technical, Operational, and Economic requirements for digital preservation in our environment

•  Ideas based on OAIS and TRAC •  Also discusses

–  Some economic and business models for sustainability

–  Case study of costs related to the development of our initial prototype


6 Strategies: Important Sustainability Issues •  Saving bits is not enough; need to be discoverable,

understandable, and playable •  Need to go beyond the capabilities of a DAM or CMS,

ensuring that we have: –  sound technical infrastructure –  good descriptive metadata –  well articulated policies –  a mission statement supporting that content

•  Interoperating factors constantly at play throughout the lifecycle, and need to be periodically revisited


3/9/11

9

Six Sustainability Strategies- •  alignment of stakeholders •  integration of archival-friendly practices in the

digital production workflow •  following OAIS and TRAC guidelines •  the application of appropriate business

models •  clear communication of value •  ongoing re-evaluation of preservation needs

and users expectations Besser, PDPTV, Austin, 3/10/11

Define and Align Stakeholder Roles and Responsibilities (1/2)

•  No organization or person was previously responsible for preservation; public and corporate libraries only received a fraction of what was produced (only content that remained late in its life-cycle)


Define and Align Stakeholder Roles and Responsibilities (2/2)

•  All stakeholders in the public broadcasting community (producers, broadcasters, viewers, rights-holders, librarians/archivists, documentary-makers) need to work together to answer these questions: –  Who has the incentive to preserve? –  Who can take on the role of preservation? –  At what point in the lifecycle of the digital work do handoffs

take place? –  Who decides what is preserved (given the enormous amount

of material created for each production)? –  Who is responsible for providing descriptive metadata? –  Who can use the materials, and under what conditions?


Integrate Archival-Friendly Practices into the Production Workflow

•  Because we were dealing with many different entities (production units, local stations, national distributors), there was little consistency of metadata, file-naming, or even of compression ratios/image quality

•  First we developed workflows that allowed the new NYU Repository to collect the “best copy” from every source (in 8 different video file formats; metadata exports from 4 diverse DBs)-


Integrate Archival-Friendly Practices NYU collects from multiple sources/qualities/metadata


Integrate Archival-Friendly Practices Phase I Workflow and Mapping

very complex


3/9/11

10

Integrate Archival-Friendly Practices Phase I Workflow and Mapping very complex






•  This obviously was not scalable •  But in 2008 a nightly news show

launched, and we were able to institute good practices around standards and metadata, making these part of their workflow process

•  This resulted in much simpler work at the Repository end-



Integrate Archival-Friendly Practices Phase II Workflow and Mapping very simple


Follow OAIS and TRAC Guidelines for Digital Preservation

•  Trustworthy Repositories Audit and Certification: Criteria and Checklist (TRAC)

•  Digital Repository Audit Method Based on Risk Assessment (DRAMBORA) toolkit

•  Bit Preservation •  Content Accessibility

–  Identified and located –  Interpreted or played back –  Delivered to users

•  Organizational Infrastructure Besser, PDPTV, Austin, 3/10/11

3/9/11

11

Identify and apply appropriate organizational & business models

(old models won’t do)

•  Business models we used in analog world are unsustainable (grant funds, one-time expenses) –  Too often we’ve seen projects obtaining funds to

digitize, but not funds to store them, add metadata, build infrastructure, make them available on Web

•  Digital gives us opportunities to increase access, create new value, and enable people to find new uses for content



(need to diversity funding sources)

•  “A sustainable project covers its operating costs through a combination of revenue sources and cost-management strategies and continues to enhance its value based on the needs of the user community.”

-Sustaining Digital Resources: An On-the-Ground View of Projects Today, Ithaka Strategy and Research, 2009



(our first set of internal tasks)

•  Identify the interests, motivations, and roles of stakeholders involved, and how they are related to each other

•  Use this to build business models •  Example

–  predict how an entity, such as a program producer or a repository, might choose to allocate its resources under given economic constraints, and indicate what actions are required by other entities, such as users or public funders, to ensure that preservation occurs


Clearly Communicate Value of Preservation and Ongoing Access to Stakeholders (1/2)

•  “To make the case for preservation, make the case for use… [After all, preservation is a] “derived demand – an activity that people undertake in the service of something else they value. In the case of digital information, people care about the possibility of future access and use, and preservation creates that potential.”

-Sustainable Economics for a Digital Planet, Blue Ribbon Task Force (BRTF) on Sustainable Digital Preservation and Access


Clearly Communicate Value of Preservation and Ongoing Access to Stakeholders (2/2)

•  Stakeholders already implicitly know that they value the results of preservation –  Producers can potentially reuse materials for new

productions –  Distributors can reach new audiences and markets –  A university professor teaching a course on human rights

can show footage of actual events to spark student interest –  The public often likes to re-watch old favorite programs

•  But they need to see how a repository can facilitate their own downstream use, and be willing to make efforts to making that repository work (submissions, metadata, cooperation, ...)


Continually re-evaluate workflows, preservation requirements, and user needs (1/2)

•  At various points in the project, we have asked a variety of stake-holders access questions that we needed to shape the system –  What type of material do they want access to? –  What different uses can they think of? –  What quality and format of video and audio is

needed? –  What metadata is most important?


3/9/11

12

Continually re-evaluate workflows, preservation requirements, and user needs (2/2)

•  In the 6 years of this project, we’ve repeatedly needed to re-examine these issues, and we need to plan on that continuing far into the future, particularly as user expectations change from the other information resources that they use

Besser, PDPTV, Austin, 3/10/11 Besser, PDPTV, Austin, 3/10/11

We were already collaborating to solve shared technical problems

•  The public television partners have been working together for a long time on common issues -- such as digital asset management, and a metadata dictionary.

•  Digitally-produced programs are at great risk of being lost --- because of the rapid changes in technology, lack of funds, and no preservation mandate.

•  Expanding our efforts to encompass preservation was a natural extension of our progress in standardizing a complex digital production and broadcast environment.


Creating Partnerships

•  It is in everyone’s interest to see that the content is saved.

•  Changes to the production workflow should be minimal and not disruptive.

•  Implementation of DAM is the perfect time to begin pushing for changes in metadata delivery.

•  This should provide immediate benefits to the producer’s work, as information and assets will be easier to find and manage.



WGBH AS A MODEL: •  At production startup, the station’s archive provides templates and

check lists for eventual program and metadata delivery.

•  Production units must deliver completed templates and all related materials before final payment.

•  Production team completes File Maker Pro databases (using templates supplied by archive) for original footage, stock footage, stills, and materials used in the final edit.



WGBH MODEL… •  Metadata collected not enough for digital preservation,

but important lessons have been learned: 1.  Gathering metadata from one system and trying to use it for

maintenance on another introduces QC and conversion problems (e.g. Users and archives have different version of File Maker Pro).

2.  Extensive QC required because metadata coming from production units is often incomplete, inconsistent, or incorrect. Content creators need to be trained as well as convinced of the importance of metadata.



CONVINCING CONTENT CREATORS… •  Create a clear financial incentive. DAMs save money (on stock

footage and shooting) when editors and producers can search and retrieve station-owned assets right from their desk.

•  Mandate from funding agencies, such as Corporation for Public Broadcasting (CPB) that preservation planning must be part of the initial proposal –  This is already happening in the sciences. NSF and NIH require long-term

data sharing and preservation plans in research proposals.

3/9/11

13

But ongoing Partnerships require both individual & institutional support •  PBS was always a “junior” partner because of

lukewarm support at the highest administrative levels

•  It is difficult to maintain collaboration with an organization facing shifting priorities –  Partnership with WNET rapidly eroded after they

dismissed their COO (even though a handful of other people at the station had been important ongoing partners)


Disappearing Partner Support


American Archive


American Archive •  Was authorized in 1967 Public Broadcasting Act, to

create an archive for public radio and television programs

•  Because of pressure around PDPTV activities, CPB finally funded beginning mid-2008

•  Manager hired and began work 2010 •  Initial prototype of material from 2 dozen stations •  Out-sourced contracts:

–  Further development and use of PB Core (2.0) –  Strategies (technology implementation plan) –  Systemwide Inventory


American Archive

•  Faces full set of CPB problems –  Facing constant PR battle over funding –  Must constantly balance service btwn television

and radio, btwn urban and rural, btwn large and small stations

–  Too many high-level administrative viewpoints on strategies and priorities

–  Almost all projects handled as outside contracts –  Too many shifts on AA (who gets contracts, focus

on preservation vs access, revising contracts mid-way, etc.)


Preserving Digital Public Television

http://dlib.nyu.edu/pdptv/

http://besser.tsoa.nyu.edu/howard/Talks/


Documents

pdptv-austin-ischoolbesser.tsoa.nyu.edu/howard/Talks/11pdptv-austin-ischool.pdf · Examined Potential Points of Metadata Capture Besser, PDPTV, Austin, 3/10/11 Examined Potential