Researching Fedora, Fedora Meeting Copenhagen 09/28/05

Building An Use Case Implementation With Building An Use Case Implementation With Denmark’s Broadcasting Archive Of Radio And Television Denmark’s Broadcasting Archive Of Radio And Television (BART)(BART)

Researching Fedora To Serve As Central Repository For Denmark’s State And University Library

Researching Fedora, Fedora Meeting Copenhagen 09/28/05

Stephan Drescher, [email protected]

picture 1 Analog Antenna Set


That’s where it all started …In an attic…

Options for System

• complete outsourcing of taskcomplete outsourcing of task• from shelffrom shelf• taylormade systemtaylormade system• own developmentown development

– around black box around black box professional equipmentprofessional equipment

– high-end consumer + Linux


Nature of the data harvest process

• 24/7/36524/7/365• single shotsingle shot• large quantities of data (220 GB a day)large quantities of data (220 GB a day)• several independent input sourcesseveral independent input sources• no unique identifier in or between no unique identifier in or between

sourcessources• different MIME types different MIME types • data needs to be evaluated and data needs to be evaluated and

eventually corrected after 48 hourseventually corrected after 48 hours• automatically ingested into repository


figure 1 Where will be FedoraFedora within Bart’s resource workflow?

TV/FM Analog SignalTV/FM Digital Signal

Footage

DescriptiveTechnicalLogistical

Metadata

Data Capture

Encoded StreamsImagesTrailers

Footage

DCMETSRDF

FOXML

Metadata

Ingester

Data Collection

Requirements...

Footage

Requirements...

Metadata

Migrate...

Dissemminator Index and Search

Data Preserve

Time MachineA/V Clippings

Footage

RelationCollection

Metadata

Disseminator DRMUser Groups

Distribute

BART


Hauppauge WinTV

Hauppauge WinTV

Hauppauge WinTV

DR1

DR2

TV2

Hauppauge WinTV

Hauppauge WinTV

Hauppauge WinTV

DR1

DR2

TV2

Hauppauge WinTV

Hauppauge WinTV

Hauppauge WinTV

DR P1

DR P2

DR P3

PC #1

PC #1’

PC #2

EMCmass-

storage

lokaldisk

lokaldisk

lokaldisk

Hauppauge WinTV

Hauppauge WinTV

Hauppauge WinTV

DR P1

DR P2

DR P3

PC #2’

lokaldisk

figure 2 Setup Capture



Encoder setup

• 3 x MPEG1/2 hardware encoder = 3 x MPEG1/2 hardware encoder = control quality and amount of datacontrol quality and amount of data

• PCM output = lossless audio, but PCM output = lossless audio, but down-sampleddown-sampled

• both FM Radio and TV frequency both FM Radio and TV frequency

tuning on a single card = flexible tuning on a single card = flexible fallback configurationfallback configuration

• parallel setup =parallel setup =

backup from original signal sourcebackup from original signal source

picture 2 Encoder And Sampler


digital resources factory… basement

Mass storage specs

• 80 - 100 TB a year (average 220 GB 80 - 100 TB a year (average 220 GB a day)a day)

• Backup on tapeBackup on tape


figure 3 Mass Storage


The cooled place…

figure 3 Signal coverage

KildeKanalkode Frekvens Format

GB/dag Start End

Landsd¾kkende tv-kanaler:DR 1 antenne 8 196.25 MPEG-2 54.8 08:00 03:00DR 2 kabel 9 203.25 MPEG-2 42.0 14:00 02:00TV2 Danmark, inkl. TV2 Æstjylland antenne 26 511.25 MPEG-2 60.0 06:00 04:00TV2 Zulu kabel 67 839.25 MPEG-1 13.5 06:45 04:30TV2 Charlie kabel 40 623.25 MPEG-1 8.1 12:45 02:00dk4 kabel 63 807.25 MPEG-1 16.2 00:00 00:00Regionale tv-kanaler:TV2 Lorry antenne (Ballerup)53 727.25 MPEG-1 3.4 11:00 12:30

18:10 23:30TV2 Syd antenne 30 543.25 MPEG-1 3.4 11:00 12:30

18:10 23:30Landsd¾kkende radiokanaler:P1 antenne 88.1 PCM 1.8 00:00 00:00P2 (Musik & Kultur) antenne 103.0 PCM 1.8 00:00 00:00P3 antenne 91.7 PCM 1.8 00:00 00:00DR Klassisk kabel 100.8 PCM 1.8 00:00 00:00Radio2 antenne 94.6 PCM 1.8 00:00 00:00Sky Radio antenne 88.6 PCM 1.8 00:00 00:00(reserveret) antenne / kabel PCM 1.8Regionale radiokanaler:DR P4 Kæbenhavns Radio antenne (Ballerup) 96.5 PCM 0.6 06:00 00:00DR P4 Radio Syd (ikke mulig) PCM 0.6 06:00 00:00DR P4 Æstjyllands Radio antenne 95.5 PCM 1.4 06:00 00:00

total (uden uge 46) 216.7


Create recordings order• Failover definitionsFailover definitions• Channel definitionsChannel definitions• Virtual encoder definitionVirtual encoder definition• Compression definitionsCompression definitions• Source definitionsSource definitions• Physical encoder definitionPhysical encoder definition• Valid Compression per physical encoder typeValid Compression per physical encoder type• Encoder mappingsEncoder mappings

– Virtual encoderVirtual encoder– Physical encoderPhysical encoder

• Abstract schedule definitionAbstract schedule definition• Query definitions (where to get start/stop info)Query definitions (where to get start/stop info)


Metadata

• Descriptive metadata Descriptive metadata – program informationprogram information– creator, author --> Dublin Corecreator, author --> Dublin Core

• Technical metadataTechnical metadata– encoding formatencoding format– create/modified dates --> METScreate/modified dates --> METS

• Logistical data Logistical data – addressesaddresses– Path to footage e.g. --> METSPath to footage e.g. --> METS


Collect metadata

• Ritzau Media Service, (MSSQL Ritzau Media Service, (MSSQL database)database)

• DR 1, (XML stream)DR 1, (XML stream)• Logfiles recording, (filesystem)Logfiles recording, (filesystem)• Logfiles storage, migration Logfiles storage, migration

(filesystem)(filesystem)


Automatic harvest and ingest system


Scheduler application

• ModulesModules– BouncerBouncer– HarvesterHarvester– LogSnifferLogSniffer– RecorderRecorder– TransformerTransformer– InspectorInspector– IngesterIngester– MigratorMigrator– SyncronizerSyncronizer


e.g. Virtual encoder definition in Recorder module


figure 4 Main Class Table

Bouncer Access to system and modules

Login, registration

Dispatcher

Harvester Collects metadata

Harvest Definition

Syncronizer,LogSniffer

LogSniffer Knows how to parse log files

List of name-value

Recorder

Recorder Schedules encoder

Recording Definition

Syncronizer

Transformer Arrange dataInto different formats, locations

e.g. From SQLResultSets toXML, METS, HTML

METSWriter,Connector

Inspector Data quality control

Checksum, Scene detection

LogSniffer

Ingester Feed repository

Upload fileXML stream

Transformer,Fedora

Migrator Move repository

Download fileXML Stream

Fedora,Transformer

Syncronizer Tune a task Mirror DatabasesCorrect Data

Transformer


Thank you!

Documents

Researching Fedora, Fedora Meeting Copenhagen 09/28/05