Embed Size (px)
Citation preview
Digital Humanities Laband the Challenges of Digital
Preservation Digital Humanities Lab
Dr. Peter [email protected]
Our Origin:Scientific Photography
Early Research
Digital Color-Reconstruction
Fast Aging
Original
Chemical Reconstruction
MathematicalModel
Digital Reconstruction
New Slide Digital Image
Faded Image
Scanning
Why Digitize
http://blog.1000memories.com/94-number-of-photos-ever-taken-digital-and-analog-in-shoebox
Digital Collections
http://blog.1000memories.com/94-number-of-photos-ever-taken-digital-and-analog-in-shoebox
6 Billion images are uploaded each month
Infrastructure
We must change to the digital domain or we loose information.
Film-Production
11
Digital data gives many new possibilities:
• lossless copies and dissemination of content
• simplified search and retrieval of content
• interlinkage of various digital sources
• … but,
• 100 hours of video are uploaded to YouTube every minute
• 250,000 photo are uploaded per minute to Facebook
• …
Consequence:
• IT grows, we face a continuous increase of digital data that needs to be preserved
Other advantages in the digital domain
a) computational methodsb) preserve informationc) accessd) lockss
Access
- Analoge assets must be stored cold, dry, dark and without much use to be preserved.
- Digital objects allow easy access, they can be shared and distributed.
Copy 1 Copy 2 Copy 3 Copy 4 Copy n
New Methods
• Computational methods to get more out of data
• Augmented objects
• Virtual Research Environments
Angle of light-source
Inte
nsity
Mathematical Model
Angle of light-source
Inte
nsity
Mathematical Model
Digital Humanities
Sca
nnin
g, bo
rn d
igita
l dat
a
Digital Humanities
Digital Preservation
• Life-cycle of digital objects depend on
- data-carriers age
- technologies of hardware change
- Software-formats become obsolete
What is digital?
• “Digital“, means: represented by discrete signsS = {s1, s2, s3, ..., sn}; n ≥2
• Every computer works digitally by using abinary set of characters with only two allowed “signs“
– S = {0,1}, S = {true,false}, S = {+ 5V,-5V}
• The alphabet we use to write text is digital as well => 26 discrete signs!
Analog <=> Digital
• Digital is – in most cases – a logical concept in which logical states are represented by a measurable physical size/parameter that can have any value in an interval.
– Magnetization on a magnetic disc (Harddrive)
– Reflectance of dyes on a coated plastic disc (CD-R)
– Electronic charge / voltage in a solid state disk
– Ink on paper
• Exceptions are: – Electron spin, quantum states
– Molecular configurations, like in DNA
– Grain in photographic film
• CD-R: 2-30 years • Tape: 30 years
• WordStar (CP/M) • DisplayWrite (IBM) • MacDraw (Apple) • PS16 (ProTracker)
• SCSI • Serial ATA • ESDI • Magneto-Optical Drive
DataCarrier
Hardware
Software
Obsolescence of Hardware
Hardware File Format
Data Carrier
• Incompatibilities and higher demands make new hardware necessary
• Life-cycle of hardware is typically 3 to 4 years
• Medium and large systems are operated with maintenance contracts
Migration means to copy the bitstream.
Data Carrier
Hardware File Format
Data Carrier
• Faster, bigger, better, cheaper • Typical live time of storage media
is in most cases longer than life-cycle of technology
• Proper storage is important (dry, cold, dark, use proper)
Migration means to copy the bitstream.
File Formats
Hardware File Format
Data Carrier
• More flexible, „modern“, more features, better for interoperability.
• If a format is obsolete it must be converted to another new format
• Unknown (lack of metadata) or obsolete file formats make data useless
Migration means to transcode the data!
30
Two approaches for archiving digital data
A) Automatic migration of a storage system with short live-cycle
DNA (Deoxyribonucleic Acid) = 2-bit Code
Coelacanthiformes is a living fossilunchanged for more than 70 million years.
Migration
• Correct migration in the computer world is time-consuming and error-prone
process.
• Consistency of data needs to be checked after copying, e.g. MD5 checksum.
• Recurrent costs with no directly added value; you have to spend money just
not to loose something.
• Depends on an efficient, technically oriented society, that takes care of its
cultural heritage. It’s an organizational problem.
• It is certainly not a „fire and forget“ approach! You have to do it.
Self Migrating Storage
Storage Storage
Storage Storage
Storage
Automatic Migration
Automatic Migration
Automatic MigrationAutomatic Migration
Automatic Migration
Automatic Migration
Automatic MigrationAutomatic Migration
B) A storage system, requiring few or no migrations at all
• Use of a permanent, stable carrier: => burnt clay
• “Visible”, the information can be seen
• The code was recognized as such=> self explaining data
Decoding might take time, but it works – in most cases. In the case of this plate the syntax & semantics has been decoded by William Henry Fox Talbot.
Analogy for archiving binary computer data
• Use of a preferably durable carrier=> Photographic Material
• Easy interface (hardware) for data recovery=> Based upon basic technology
• Application of an open and common (standardized) data format=> Based upon basic technology
Hardware Software
Data Carrier
Digital encoding
1. An arbitrary bit code (here: a TIFF file) is rendered in the form of a 2D barcode
2. A long-lasting and technology-autonomous data carrier is used
...0010101001...
Digital Encoding
AIFF
TIFF
2D Barcode
Visual binary code: Nothing new
IBM 1360 Photo-Digital Storage System
Dolby Digital:Sound data between perforation holes
Storage DocumentationDescription
METS Metadata<?xml version="1.0"?>METS_Profilexmlns="http://www.loc.gov/METS_Profile/"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://www.loc.gov/METS_Profile/http://www.peviar.ch/monolith/mets/mets.profile.v1-2.xsd"><URI LOCTYPE="URL">http://www.iml.unibas.ch/Monolith/Mets-Profile</URI><title>Monolith Mets Profile</title><abstract>This is the METS Profile used todescribe digital objects archived with Monolith.</abstract><date>2006-01-01T12:00:00</date><contact><name>N.N.</name><institution>Bitsave AG</institution><address>Bernstrasse 23, CH-3122Kehrsatz</address><email>[email protected]</email></contact><registration_info regDate="" regURI=""/><metadataxmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>Digitalisierungvon "Mona Lisa"</dc:title><dc:creator>Graf, F.Otto</dc:creator><dc:publisher>Graf DigitalLibrary</dc:publisher><dc:type>image</dc:type><dc:format>image-tiff</dc:format><dc:format>Colordepth: 8 bit</dc:format><dc:format>Color Space:AdobeRGB</dc:format><dc:identifier>CHBS_LIB99_AX01</dc:format><dc:source>FRIDF_L01_A001</dc:source><dc:rights>Copyright Graf, F.Otto</dc:rights></metadata></METS_Profile>
EAD Finding Aid<?xml version="1.0" encoding="UTF-8"?><eadxmlns="urn:isbn:1-931666-22-9"xmlns:xlink="http://www.w3.org/1999/xlink"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="urn:isbn:1-931666-22-9./ead.xsd"relatedencoding="MARC"audience="external"><eadheaderrelatedencoding="marc" langencoding="iso639-2b"scriptencoding="iso15924"repositoryencoding="iso15511"countryencoding="iso3166-1"dateencoding="iso8601"audience="internal"><eadidcountrycode="CH">graflib.ch/collections/ML</eadid><filedesc><titlestmt><titleproper>Finding aid to Mona Lisacollection</date></titleproper></titlestmt><publicationstmt><publisher>Graf, F. Otto</publisher><address><addressline>Bleicherweg9</addressline><addressline>4000 Basel</addressline></address><datetype="publication" era="ce" calendar="gregorian">Published in2006</date></publicationstmt></filedesc><profiledesc><creation>Machine-readable finding aid encoded in EAD v.1.0 on<date normal="20000509"type="encoder" era="ce" calendar="gregorian">May 18,2006</date></creation><langusage>Finding aid written in<language>English</language>.</langusage></profiledesc></eadheader><archdesc level="collection" type="inventory"><did><head>Description ofthe Collection</head><repository encodinganalog="852" label="Location ofcollection:"><corpname>Graf Library Mona LisaCollection</corpname><address>...</address></repository><abstractlabel="Short description of collection:">...</abstract><noteactuate="onrequest"><p>See below for information on access andrestrictions.</p></note><langmaterial label="LanguagesRepresented"><languagelangcode="eng">English</language></langmaterial></did><dsctype="in-depth"><head>Container List</head><c01level="subgrp"><did><unittitle>Color DaylightImages</unittitle></did><c02 level="series"><did> <unittitle>Series 1:2003</unittitle></did><c03 level="file"> ...</c03></c02></c01></dsc></archdesc></ead>
CHBS_LIB99_AX01
43
Example of Monolith 35 mm Film
Film as information carrier: The theory
PHOTOGRAPHIC SCIENCE AND ENGINEERIN Volume 6, Number 5, September-0ctober 196
pages 281·
The Application of Fourier Techniques and Information Theory to the Assessment of Photographic Image Quality
R. SHAW, Research Laboratories, !liord Limited, Brentwood, Essex, England
In the more advanced photographic systems of the type used in satellites, it is essential that the highest possible efficiency is obtained. By using the powerful tools of Shannon's Information Theory, an over-all figure of merit may be obtained for the capacity of a film to receive and store information. This involves joint consideration of the speed, contrast-transfer function, and noise power spectrum of the film. The significance and practical evaluation of this figure of merit are discussed, and examples are given for various film-developer combinations.
The Contrast-Transfer Function and the Noise Power Spectrum
In 1946 considerable impetus was given to the subject of assessing optical systems by Duffieux, l when he proposed the application of the same Fourier techniques which had already proved of great value in communication theory. Further developments along these lines were made by Elias, Grey, and Robinson, 2 Schade, 3 Elias,4 Fellgett,5 Fellgett and Linfoot,6 and Jones.7
The use of Fourier theory enables image assess-ment to be made in terms of the Fourier spectral elements of the object and image, and by so working in the spatial frequency (u,v)-plane instead of the distance (x,y) -plane, considerable simplification can be made in the description of the performance of optical and photographic systems. Because of the close relation between spatial frequency and fineness
Presented at the Annual Conference, Boston, M ass., 10 M ay 1962. Received 16 M ay 1962.
1. P. M. Duffieux, L'integrale de Fourier et ses applications a l'optique, Private publication, Rennes, 1946.
2 . P. Elias, D. S . Grey, and D. Z. Robinson, J. Opt. Soc. Am., 42: 127 (1952).
3. O. H. Schade, J. SM PTE, 58: 181 (1952). 4 . P. Elias, J. Opt. Soc . Am., 43: 229 (1953). 5. P . B. Fellgett, J. Opt. Soc. Am., 43: 271 (1953). 6. P. B . Fellgett and E. H. Linfoot, Phil. Trans . Roy. Soc. London, Ser.
A, 247: 369 (1955). 7. R. Clark Jones, J . Opt. Soc. Am., 45: 799 (1955).
354
of detail there is little loss in intuitive clarity, and today the contrast-transfer function and noise power spectrum are widely regarded as providing the most convenient means of assessing film spread and granularity.
In recent years measurements of both these func· tions have been made for many of the films which are available on the market.8 - 14
Information Theory Although manufacturers of film provide a speed
rating for each film, usually no rating of image quality is given. While the speed rating is usually a satisfactory guide for the ordinary photographer, it is insufficient when choosing a film for scientific pur-poses where the greatest possible amount of informa-tion has to be' recorded by the film. Here it may prove more than worthwhile if the efficiency can be increased by even a small amount by a judicious choice of the best film-developer combination. A
8. E. Ingelsta m , E. Djurle, and B. Sjogren, J. Opt. Soc. Am., 46: 707 (1956).
9 . R. L . Lamberts, J . Opt. Soc. Am., 49: 425 (1959); 51: 982 (1961). 10. L . O. Hendeberg, Arki" Fysik, 16: 417,457 (1960). 11. D. H . Kelly, J. Opt. Soc. Am., 50: 269 (1960); 51: 319 (1961). 12. M . Tamura and H . Kubota, J. Appl. Phys. (Japan), 26: 92 (1957). 13. H . Ohzu and H. Kubota, J . Appl. Phys. (Japan), 26: 96 (1957). 14. H. Frieser, Mitt. Forschungslab. Agfa Leverkusen-Muenchen, 2: 249
270 (1958) .
slow film overdeveloped may well yield better results than a fast film developed normally, when both are used for the same purpose. However, no indica-tion of this can be gained from knowledge of their respective speed ratings.
If the contrast-transfer function and noise power spectrum of each available film are known, as well as the speed, the choice of the most suitable film may be simplified. But, although both play an essential part in determining image quality, the extent to which each does so is not immediately obvious. Further, both may vary considerably and in a sub-stantially different manner from film to film, and also will depend on the conditions of exposure and de-velopment.
These problems arise whenever a manufacturer attempts to improve one of his films. An improve-ment in some respects will often involve a loss in others, and it is difficult to decide whether or not an over-all improvement has been made. For example, a slight increase in speed may be of doubtful value if it is at the expense of a large increase in granularity. Clearly what is needed is some over-all figure of merit which can take into account both the image quality (determined by the contrast-transfer function and noise power spectrum) and the efficiency with which it is obtained (determined by the speed ).
In 1948 C.E.Shannon l5 developed a mathematical technique of crucial importance in this connection, and the name Information Theory is now generally used to describe his results. Shannon showed that there is only one way to define a measure of amount of information which is consistent with basic in-tuitive notions. He further showed how, for certain types of communication system, the information conveyed about a given signal can be calculated from considerations of signal-to-noise ratio. A convenient unit of information is the binary digit, or "bit," which corresponds to the 'amount of in-formation given by a "yes" or "no" answer when both were initially equiprobable.
The possible application of Information Theory to optical and photographic systems began to attract attention around 1953, both the work of Blanc-La-pierre l6 and Fellgett and Linfoot6 being prominent. It was in fact Fellgett and Linfoot6 (1955) who arrived at the first formulation of an informational figure of merit for a photographic film involving both the con-trast-transfer function and the noise power spectrum. Recent developments in the practical application of this figure of merit have been made by Linfootl7 and Jones. l8*
Formulation Fellgett and Linfoot showed that for a random
object intensity distribution (under the constraint
• I a m gra teful to b oth D r. Linfoot and Dr. Jones for a llowing me t o see their pa pers in advance of publication. 15. C. E. Sha nnon , Bell System Tech . J ., 27: 379, 623 (1948) . 16. A Blanc-Lapierre, Symposium on M icrowave Optics, 2: No . 46,
McGill Univ., 1953. 17. E . H . Linfoot, J . Phot. S ci. , 9: 188 (1961). 18. R. Clark Jones, J. Opt. Soc. Am., 51: 1159 (1961).
of a prescribed statistical-mean total spectral den-sity) the mean information content recorded in the emulsion area B, at the mean density level D, is given by
+'" 1= B II log! (1 + S (u,V» du dv bits/ image (11
where the signal-to-noise power ratio S (u ,v ) is given by
S (u v ) = T 2(U,V )'p (u,v ) , n (u,v ) (2)
Here, p (u,v ) is the spectral power density of the fractional fluctuations in the objectt intensity distribution, n (u,v ) is the spectral power density of the photographic noise at the mean recording level D, and T (u,v ) is the photographic contrast-transfer function at this level. Both T (u,v ) and n (u,v ) will be those values corresponding to the particular ex-posure-film-developer combination, but p (u ,v) is independent of film properties.
Informational assessments of photographic image quality relative to those random object distributions for which p (u,v ) is constant as (u,v ) varies form the simplest analytical approach. In practice this corresponds to selecting a film by means of an in-formational rating when initially there is a complete lack of knowledge about the spatial distribution of the object. Also, to comply with the constraint on the total spectral density as previously mentioned, the information content is in the first place evaluated for a finite spatial frequency range. This is mathe-matically equivalent to assuming that p (u,v ) = 0 for all (u ,v ) outside some circle u 2 + v 2 < H 2 in the spatial frequency plane, and then letting H tend to infinity. As soon as the circle includes the whole of the spatial frequency range in which T (u,v ) is effectively nonzero, the information content remains unchanged by a further increase in H,
Because of the isotropic properties of photographic film, measurements taken in one direction are suf-ficient to provide a complete description of film characteristics. Hence, it is convenient to use the line frequency w, defined by w 2 = U 2 + v 2, and after making this substitution we obtain
I = 21T i H log2 (1 + w dw bits/ unit area
(3 )
where p o is the constant value of p ew) within the range 0 <w <H .
Most methods of measurement yield the normal-ized contrast-transfer function defined by
t For simplicity the object is regarded h ere as the intensity distribution a rriving a t the surface of t he film . At this stage i t will usually be modi-fied a lready by such fact ors as the optical-tra nsfe r functi on of the camera sys te m .
355
“Sharpness“ of film
“Sharpness“ of machinery
Noise of film (grain)Storage Capacity
C. E. Shannon, Bell System Tech.J ., 27: 379, 623 (1948).
Area of 24 x 18 mm2 on analogue motion picture film is capable to store ~ 0,8 - 1,2 MByte!
Resulting Capacity for Film
Typical color film material will allow to store < 1 MByte per frame (full app) netto information!
Max. Capacity Kodak Fine Gain Cine Pos
Kodak Royal-X
Kodak Panatomic-X
Microfilm 7456
Microfilm 5454
bit/cm2 3.30E+05 1.40E+05 7.00E+05 8.80E+05 1.60E+06byte/cm2 4.13E+04 1.75E+04 8.75E+04 1.10E+05 2.00E+05Byte/FullAPP Filmframe
1.78E+05 7.56E+04 3.78E+05 4.75E+05 8.64E+05
MByte/FullAPP Filmframe
0.178 0.076 0.378 0.475 0.864
MByte/FullAPP Filmframe Approximativ for similar color material (x 2.4)
0.428 0.181 0.907 1.140 2.074
Color film can be handelt as three layers, but layers placed lower have less resolving power.
46
An photograph on film contains not more information as 0,8 - 1,2 MByte!
Workflow 72 MByte image-data 72 MByte image-data
Area of 24 x 18 mm2 on analogue motion picture film is capable to store ~ 0,8 - 1,2 MByte!
Archival File Format!
Well documented
Widely used
High quality Standardized
TIFFArchival File Format
• TIFF: Can be RGB, CMYK, 8 bps, 1bps, uncompressed, compressed, even JPEG, multipage, ….
• A TIFF file ending is not specifying what is in there!
• Private tags might render TIFFs useless
Memory institutions want to know what files they have!
TIF Format
It is there!
• TIFF was the major standard (digital master) for archival purposes • TIFF is a final rendered image (TIFF/EP is a raw!) • TIFF is 16bit • TIFF is lossless • TIFF is professional
• Many archives and museums store TIFF • because the others do it • because it is widely used and well documented • because it is promised quality
88% MemoryInstitutions
TIFF Format
Standardization
• TIFF is a complex standard with a lot of possible features. It is not clear to say what “correctness“ means. It can depend on the application.
• Therefore it’s necessary to define and recommend a well-defined set of features (tags) that are accepted by most memory institutions and are appropriate regarding sustainability.
TI/A Standard Initiative
Recommendation of TIFF for Archives
Well specified set of mandatory, forbidden and optional tags
Standardization process in cooperation with ISO TC171, the PDF group
The approach is well-known from Portable Document Format: • A PDF is a full featured digital document, some of the features are not wanted in archives. • A PDF/A is an adopted version, fulfilling archival needs. Some features are forbidden, some
are required. • We follow a similar approach: TIFF will be precisely specified for the use in archives.
Recommendation
• It is important to cover most of the data in existing repositories • Migration only shall occur, if data is in danger to become obsolete
Expert Community Feature Histogram of real data
ISO-Standardized recommendation
for the use ofTIFF in Archives
Digital Humanities
Scan
ning
, bor
n di
gita
l
Arc
hivi
ng (
Sust
aina
bilit
y)
The Gap between Scanning and Archiving
• The Internet allow access to digital objects
• Repositories allow to search for digital objects
• Open Linked Data is of more and more importance=> Standardized interfaces are required
Research tools in den Humanities…
…some years ago
Joint Information Systems Committee defines
Virtual Research Environments (VRE)
• Tools & Technologies for
- Researchers
- Collaboration
- Interaction
VRE
Repository Working environment
Communication & sharing
Annotation
Linkage
Storage
Archival (longterm)
Data- service
ext. data- provider
Requirements for a Digital Research Platform (in Humanities, but also in…?
• Flexible, versatile repository
- Encoded text (TEI, UTF-8,…), music (MEI,…), facsimile, images, sound, moving image,…
- Versioning & change history ☛ citations
• Methods and tools
- Annotation & Linkage (Nanopublications)
- Analysis tools
• Presentation & Visualization
- VRE
- Visualization (2D, 3D, dynamic,…)
• Collaboration
- Communication
- Sharing
• Longevity (decades or more)
}knowledge organisation, representation & annotation
screenshot: transcription of text
WebGL 3D-model based on Merian bird view map of Basel (1615)
Diary of Felix Platter (1536-1614), doctor during the plague, describing the houses he visited
Merian map 1615
Interlinking of text, images and 3d
Graph view
And all the mentioned problems of the digital domain…
Virtual Research Environment
• Generalized approach for…
• the work with digital sources
• the visualization of context and correlation
• enriched content and meta-information (e.g. geographic data etc.)
• “knowledge mining” (semantic search, network analysis etc.)
Efficiency
• Standardized infrastructure:
- Central infrastructure simplifies digital preservation.
- The wheel (the main technology) must not be reinvented for each and every project. Resources can be focussed on maintenance and enhancement (eg dedicated plug-ins).
- Already existing tools and methods can be reused and shared with oder scholars from other disciplines.
Digital HumanitiesG
enes
e (s
cann
ing,
born
dig
ital)
Digital Humanities
Arc
hivi
ng (
Sust
aina
bilit
y)
App
licat
ion
(sha
re, l
ink)
Use
and
life
cyc
le
man
agem
ent
of d
igita
l obj
ects
Application und Technology,the Theory
The Reality
