30 October 2014 SIPAD Workshop
Basic Principles of Digitisation – of audio, video and film
Richard WrightSOIMA-LATAM, Mexico City, October 2014
Review
Audio and Video are signalsAudiovisual Archives hold recordings of signalsA signal is described by two attributes:• Frequency range = bandwidth• Dynamic range = signal-to-noise ratioPure tones = only one frequency (tuning fork)All other sounds have a spectrum of frequenciesSpeech: 100 Hz to 10 000 Hz (= 10 kHz)
30 October 2014 SIPAD Workshop
Review
All other sounds have a spectrum of frequencies• Speech: 100 Hz to 10 000 Hz (= 10 kHz)• Music: 30 Hz to above 18 000 Hz• Human hearing (normal): 50 Hz to 18 kHz• Telephone bandwidth: 300 Hz to 3.3 kHz (old)• 78 rpm recording (shellac): up to 8 kHz• 33 1/3 (vinyl; LP): up to 15 kHz (hi-fi)• Audio tape (1/4”): up to 18 kHz (at 15 ips; many
different kinds of tape, and bandwidth varies with recording speed)
• CD: theoretical max= 22 kHz = (44.1)/2; practical max = 17.6 kHz = (44.1)/(2.5)
30 October 2014 SIPAD Workshop
Dynamic Range
• Speech: typically 30 to 50 dB; max about 70 dB• Music: could be 70 dB in concert hall
– Could be 90 dB in a very quiet studio
• Human hearing: 100 dB without risk of damage; 130 dB max
• Old telephones: 20 to 30 dB• 78 rpm shellac recording: 40 to 50 dB• Sound on film: typically 40 to 50 dB• 33 1/3 rpm vinyl LP: 60 to 70 dB• ¼” audio tape: just over 70 dB (best = 73 dB)• CD: capable of over 90 dB range (96 dB in theory)
30 October 2014 SIPAD Workshop
5
The need for digitisation
Preservation planningmapping your collections;setting priorities;making a collection strategy,a preservation strategyand a preservation plan
The Preservation Factory approach
Conservation, Digitisation and Preservation
Terminology
Preservation – “Everything needed to ensure permanent access” = maintenance Conservation: keeping what you have (for as long
as you can) = safe storage and handling Preservation actions: interventions. Changing what
you have = repair and replace− Making a new negative or interneg− Copying from an old carrier to a new carrier− Digitising
Which separates content from carrier = liberation
6
Conservation
packaging, handling and shelving environmental conditions: temperature and
humidity; protection from pollution, dirt etc protecting the masters; and condition monitoring = checking the stock
Full description on Preservation Guide wiki: http://wiki.prestospace.org/pmwiki.php?n=Main.PreservationStrategy#Conservation
7
8
The need for digitisation
Only three ways to preserve:
1.Keep what you have
2.Copy using same technology
3.Copy using new technology
Keep what you have: only works for film• Audio, Video: analogue technology obsolete
Copy on same technology: only works for film• Audio, Video: analogue technology obsolete
9
What about vinyl?
Gramophone records: lacquer, shellac and vinyl Lacquer = a master
recording (acetate, instantaneous disc); very fragile !
10
More on gramophone records
Shellac was used for 78 rpm commercial recordings
Also fragile – the main risk is handling; the shellac itself is stable (compared to laquer)
11
But, what about vinyl?
Used for 45 and 33 1/3 rpm recordings They are fragile (though less fragile than
shellac) Warp from heat (can be fixed with care) Easily scratched, and can get very dirty The groove can be damaged in playback (if the
needle and tone arm adjustment is not right) Vinyl is soft !!! Vinyl and shellac can suffer chemical damage
Vietnam Film Institute Workshops 12
Gramophone damage
Dropping a needle onto a vinyl disc
Oil coming to the surface on a lacquer disc
13
Where was I?
Only three ways to preserve:
Keep what you have: only works for film• Audio, Video: analogue technology obsolete
Copy on same technology:only works for film• Audio, Video: analogue technology obsolete
So we are left with only one option (for audio and video):
Copy using new technology
14
Preservation of Film
Many excellent example of conservation Film technology still exists (but in decline) BUT – the situation is rapidly changing
Kodak in severe economic trouble Commercial cinemas changing to digital projection
− Norway changed completely in 2011-2012
Commercial cinema will change or go out of business
Commercial cinema will not keep old projectors
15
The end of film projection?
“ We're About to Lose 1,000 Small Theaters That Can't Convert to Digital. Does It Matter?”Indiewire, USA 2012
16
Digitisation of Film
Now: needed for access Internet Digital cinema
Soon: needed for preservation When manufacture of film stock is stopped
Result: ALL media needs to be digitised, audio and video and film That's a lot of content that has to be digitised There isn't enough time There isn't enough money
17
PRESTO Digitisation
“Better Faster Cheaper” Daniel Teruggi, Institut National de l'Audiovisuel (INA), Paris
Basic concepts:
1) Dealing with the whole collection
2) Developing a strategy For the institution and its collection For the preservation work
3) Developing a preservation plan
18
Digitisation Planning
All the Presto information on a wiki: www.preservationguide.co.uk/RDWiki/pmwiki.php
StrategyStrategy – what does your institution do? What does the collection do? What can digitisation do?
PlanningPlanning – making a preservation plan How to estimate a budget Building a business case
19
Collection Strategy
Long-term purpose of the digitisation Preserving the collection Maintaining services (old business) Creating new services (new business)
Physical Outcomes: Digital files Mass storage Cheaper, better maintenance Cheaper, better digital access copies Web access
20
Access Outcomes
Collection access from computer stations Mediatheque approach
Collection access via the Internet With restricted access
− British Film Institute: higher education, public libraries … and on YouTube
− British Library Sound Archive: higher education institutions only
Or even unrestricted access− INA has 30 000 hours of broadcast content online !
21
Restrictions on Access
Attitudes: we don't do that Fear: somebody might complain Rights: we don't hold the rights
A solution:
1) public institutions create public value by opening their collections as widely as possible
2) non-fiction content has the most information and the least rights problems
We have a public service obligation to create access to our collections !
22
Access:Technical Requirements
A usable catalogue Online support services: questions, linking to
online sales, dealing with faults Logins, access control, authentication, data
protection … Computing power to support what could be
large numbers of people requiring access UK National Archive crashed; Europeana crashed
Managing lots of information technology
23
Digital Archive: workflow
Everything changes when shelves disappear: Acquisition: digital ingest Cataloguing: complex metadata issues: embedded
metadata, preservation metadata, mapping Curation: huge opportunity to create online
collections Research: self-research, fast scanning of 'hits' –
changes the requirements of cataloguing Access: requirements need to be built into the
digitisation workflow (access copies, public metadata, rights clearance, censorship?)
24
Getting Started
Map – The collection needs to be divided by format, condition, purpose ...
Priorities – arrange the “areas on the map” according to what needs to be done first
wiki: Make a Map of your CollectionDivide the collection by physical formats, and collect the following information on each format:
age rangestorage historygenre or valuephysical condition
Getting Started
25
BBC film examples from wiki
Map: http://www.preservationguide.co.uk/RDWiki/pmwiki.php?n=Main.GettingStarted
Strategy: http://www.preservationguide.co.uk/RDWiki/pmwiki.php?n=Main.PreservationStrategy
Plan: http://www.preservationguide.co.uk/RDWiki/pmwiki.php?n=Main.PresPlan
Also: tutorials on PrestoCentre websitecollection strategy and preservation plans_V1.01_0.pdfmapping your collection_V1.01_0.pdf
Collection Strategy & Plan
• PDFs of wiki ‘chapters’
– Collection Strategy
– Preservation Strategy
– Preservation Plan and Budget– Collection Strategy Preservation Strategy
– Preservation Plan and Budget
• Tutorials on PrestoCentre– collection strategy and preservation plans_V1.01_0.pdf
– mapping your collection_V1.01_0.pdf
30 October 2014 SIPAD Workshop
27
The Factory Approach
The division of labour in pin manufacturing:(and the great increase in the quantity of work that results)
The division of labour in pin manufacturing: (and the great increase in the quantity of work that results)
Approach: NOT about cutting corners or reducing quality
Instead: about cutting wasted time and wasted effort
Batches: doing one thing at a time, and then doing it again (and again, and again)
Problem: how to keep the work interesting! 28
The Factory Approach
29
A BBC Preservation Factory
30
The Factory Approach
“Division of labour” requires having specialists, which means having a team
Which means having a middle-sized or larger project
But: even a single person can use a batch approach, and get some efficiencies
BBC Preservation:£5M per year (for 20 years)
Audio: 7000 hours per year of ¼ inch tapeAlso digitised 63 000 “sepmag” elements: separate magnetic sound tracks for film;− Transferred to CD (good!) and to polyester (bad!)
Also 35 000 shellac and vinyl recordingsAlso DAT and MD digital recordings
2” videotape: 46 000 tapes, transferred to D3 or Digibeta, done over 7 years
1” videotape: 80 000 in 5 yrs, to D3/Digibeta U-matic: 40 000 in 3 yrs, to MPEG-2 files
32
News film: 43 000 items in 3 years Ektachrome; digitised to SD video (Digibeta)
10% of total cost is cataloguing 20 to 30% is quality control (checking) Since 2008: transfer 40k D3 videotape to files:
Uncompressed; MXF wrapper = INGEX Now: transfer of BetaSP and Digibeta to files Major confusion over High Definition formats !
33
Discussion
Digitisation projects of delegates Audio, video, film Which physical formats (eg ¼-inch audio tape,
1-inch video tape, 16mm B&W film … ) ? Size: how much content, how many people,
how many years ?
34
Time for a break
After the break – a choice !
1)Low-level: standards and best practice for audio, video and film
2)Higher level: case studies for preservation and access
3) Your questions on digitisation your long-term plan? your immediate problems?
30 October 2014 SIPAD Workshop
Digitisation Issues:
• Audio and Video: no problems at all with the digitisation technology – the problems are with:– Playback of originals
– Standards: compression, formats
Audio holdings (TAPE data)Carrier %
wax cylinders 0.03
coarse groove replicated disks (‘78s’,‘shellacs’)
1.9
instantaneous disks of any kind 0.2
microgroove disks (LPs) 17.5
open reel magnetic tape 38.6
compact cassettes 20.6
R-DAT 2.5
replicated CDs, DVDs 16.5
recordable and rewritable CDs, DVDs 1.4
Minidisks 0.4
Other 0.3
Total 100
Audio: major problems• Wax cylinders: condition, equipment
• Lacquer discs: condition
• Sticky shed on 6mm tape playback
• DAT recordings: technical standards, condition, equipment
• CD ‘ripping’ – common issue for digital-digital transfer: error concealment (and what to do about it)
• IASA TC-04: Guidelines on the Production and Preservation of Digital Audio Objects, March 2009.
– BWAV, 96kHz, “24bit”; BBC do 44.1kHz, 16bit
How to digitise audio
• IASA TC-04
• 48 kHz, “24 bit”– 96 kHz recommended
• Save as a wav file
• Broadcast Wave Format recommended
Film: major problems
• No problem with playback equipment !
• Some film can last centuries
• Some film already destroyed
vinegar syndrome; colour fade; mold
• Telecine out, datacine in
Datacine for film digitisation
• Batch process• HD or better for 16mm• Many archives doing 2K scanning• No grading shot-by-shot– so a higher dynamic range is needed: 14 bits or better• Needs sprocketless transport to cope with fragile film,
shrinkage, splices• Infrared technology for scratch detection• Wet-gate for scratch concealment (on one side)• Several new sprocketless systems, €80 to 240k
How to Digitise Film
B&G document on PrestoCentre
http://www.prestocentre.eu/sites/dev.prestocentre.antenna.nl/files/filmscanning_v1.03.pdf
White Paper: Film scanning considerations
By Tom de Smet and Harm Jan Triemstra
Sound on Film
• Successful digital processing of optical sound tracks now very well established by Chase Audio, Hollywood– But expensive
• Analogue capture most cost-effective if sound track in reasonable condition
• Much TV archive material has separate sound tracks: sepopt, sepmag (beware vinegar!)
Video: major problems
• As for audio: equipment, operators, fragile and decaying media; sticky-shed
• Plus: time base correction and dropout ‘concealment’ not built into old equipment
• Colour: composite signals and their decoding;
– BBC: a composite original is PAL-decoded and saved as component, even though “the original” is composite
• Head wear, head life, head replacement – alternative technology for reading videotape has been propose
Video digitisation issues
• What codec, what file format?– Uncompressed, lossless, lossy
– Lossless: JPEG2000, FFV1 (many others)
– MOV, AVI, MXF ... (cf LOC FADGI case studies)
• Endgame: uncompressed (no codec !)
• BBC: saving uncompressed from D3 transfers to files: INGEX open-source software (as used by Tate Galleries, London)
How to digitise video
1. Save the original2. Digitise @ SDI = 4:2:2 = 200 megabits/sec3. Save exactly as digitised = uncompressed4. Use an open source file format (MXF or ?)
3b – or lossless = lossless JPEG2000, 3c – or ‘mezzanine’ = high-quality lossy
(but only for low quality originals)(digibeta is a ‘mezzanine’ format)
Common issue: quality control
• Computer processing exists– in professional audio equipment: Quadrega, NOA …
• ‘fault detection’ technology from restoration work
• Joanneum video problem detector = VidiCert
• Audio Inspector https://www.audioinspector.com/
• New QC software from Dave Rice, BAVC http://www.bavc.org/qctools
• Key issues: workflow integration, statistics
How to Digitise Film
• B&G document on PrestoCentre
• http://www.prestocentre.eu/sites/dev.prestocentre.antenna.nl/files/filmscanning_v1.03.pdf
White Paper: Film scanning considerations
By Tom de Smet and Harm Jan Triemstra
Recommended